CONVEX FUNCTIONS AND OPTIMIZATION TECHINIQUES A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF

Transcription

1 CONVEX FUNCTIONS AND OPTIMIZATION TECHINIQUES A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE IN MATHEMATICS SUBMITTED TO NATIONAL INSTITUTE OF TECHNOLOGY, ROURKELA BY ARUN KUMAR GUPTA ROLL NO. 409MA2066 UNDER THE SUPERVISION OF Dr. ANIL KUMAR DEPARTMENT OF MATHEMATICS NATIONAL INSTITUTE OF TECHNOLOGY ROURKELA

2 NATIONAL INSTITUTE OF TECHNOLOGY ROURKELA DECLARATION I hereby certify that the work which is being presented in the thesis entitled Convex function and optimization techniques in partial fulfillment of the requirement for the award of the degree of Master of Science, submitted in the Department of Mathematics, National Institute of Technology, Rourkela is an authentic record of my own work carried out under the supervision of Dr. Anil Kumar. The matter embodied in this thesis has not been submitted by me for the award of any other degree. Date: (Arun Kumar Gupta) This is to certify that the above statement made by the candidate is correct to the best of my knowledge. Dr. Anil Kumar Department of Mathematics National Institute of Technology Rourkela Orissa, India.

3 ACKNOWLEDGEMENTS I wish to express my deep sense of gratitude to my supervisor, Prof. Anil Kumar, Department of Mathematics, National Institute of Technology, Rourkela for his inspiring guidance and assistance in the preparation of this thesis. I am grateful to Prof. P.C. Panda, Director, National Institute of Technology, Rourkela for providing excellent facilities in the Institute for carrying out research. I also take the opportunity to acknowledge quite explicitly with gratitude my debt to the Head of the Department, Prof G.K. Panda and all the Professors and Staff, Department of Mathematics, National Institute of Technology, Rourkela for their encouragement and valuable suggestions during the preparation of this thesis. I am extremely grateful to my parents, and my friends, who is a constant source of inspiration for me. (ARUN KUMAR GUPTA)

4 TABLE OF CONTENTS Chapter 1 Introduction Chapter 2 Convex functions and generalizations Chapter 3 Constrained Minimization Chapter 4 Optimality conditions Chapter 5 References

5 CHAPTER -1 Introduction: Optimization is the process of maximizing or minimizing a desired objective function while satisfying the prevailing constraints. The optimization problems have two major divisions. One is linear programming problem and other is non-linear programming problem. But the modern game theory, dynamic programming problem, integer programming problem also part of the Optimization theory having wide range of application in modern science, economics and management. In the present work I tried to compare the solution of Mathematical programming problem by Graphical solution method and others as well as its theoretic descriptions. As we know that not like linear programming problem where multidimensional problems have a great deal of applications, non-linear programming problem mostly considered only in two variables. Therefore for nonlinear programming problems we have an opportunity to plot the graph in two dimensions and get a concrete graph of the solution space which will be a step ahead in its solutions. Nonlinear programming deals with the problem of optimizing an objective function in the presence of equality and inequality constraints. The development of highly efficient and robust algorithms and software for linear programming, the advent of high speed computers, and the education of managers and practitioners in regard to the advantages and profitability of mathematical modeling and analysis, have made linear programming an important tool for solving problems in diverse fields. However, many realistic problems cannot be adequately represented or approximated as a linear program owing to the nature of the nonlinearity of the objective function or the nonlinearity of any of the constraints.

6 CHAPTER -2 Convex Functions and Generalization: Convex and concave functions have many special and important properties. In this chapter, I introduce the important topics of convex and concave functions and develop some of their properties. For example, any local minimum of a convex function over a convex set is also a global minimum. These properties can be utilized in developing suitable optimality conditions and computational schemes for optimality problems that involve convex and concave functions. 2.1 Definitions and Basic properties: Convex Function: A function defined on a convex subset X of is said to be convex on X if ( ( ) ) ( ) ( ) ( ) for each and, -. The function is called strictly convex on X if the above inequality is true as a strict inequality for each and, -. Concave function: A real-valued function defined on a convex subset X of is said to be concave on X if ( ( ) ) ( ) ( ) ( ) for each and, -. The function is called strictly concave on X if the above inequality is true as a strict inequality for each and, -. Also, ( ) is concave on [a, b] if and only if the function ( ) is convex on [a, b]. Now let us consider the geometric interpretation of convex and concave functions. Let be two distinct points in the domain of, and consider the point ( ) with, -. So for a convex function, the value of at points on the line segment ( ), is less than or equal to the height of the chord joining the points, ( )- and, ( )-. For the concave function, the chord is below the function itself. Hence, a function is both convex and concave if and only if it is affine. Figure 2.1 shows some examples of convex and concave functions.

7 ( ) y ( ) y Convex function concave function x y neither convex nor concave Figure 2.1 examples of convex and concave functions The following are some examples of convex and concave functions. Examples of convex functions: 1. Any function of the form: ( ) 2. Powers: ( ) 3. Powers of absolute value: ( ) 4. Exponential: ( ) 5. Every linear transformation taking values in R is convex. 6. ( ) 7. Every norm is a convex function, by the triangle inequality and positive homogeneity. 8. ( ), -

8 Examples of Concave functions: 1. ( ) 2. ( ) 3. Any linear function ( ) 4. The function ( ) ( ) is concave on the interval, - 5. Logarithm: We give below some particularly important instances of convex functions that arise very often in practice. 1. Let, be convex functions. Then, (a) ( ) ( ), where for is a convex functions. (b) ( ) Maximum { ( ), ( ) ( ) } is a convex function. 2. Suppose than is a concave function. Let * ( ) + and define ( ) ( ) Then, is convex over S. 3. Let be a nondecreasing, univariate, convex function, and let be a convex function. Then, the composite function defined as ( ), ( )- is a convex function. 4. Let be a convex function, and let be an affine function of the form h(x) =Ax + b, where A is a matrix and b is a vector. Then, the composite function defined as ( ), ( )- is a convex function. 2.2 Epigraph and Hypograph of a function: A function on S can be described by the set *, ( )- + which is referred to as the graph of the function. We can construct two sets that are related to the graph of the epigraph, which consists of points above the graph of, and the hypograph, which consists of points below the graph of. These notions are discussed in the definition below Definition: Let S be a nonempty set in and let.the epigraph of, denoted by is a subset of defined by *( ) ( )+ And the strict epigraph of the function is denoted by *( ) ( )+

9 The hypograph of a function is denoted by is a subset of defined by *( ) ( )+ And the strict hypograph of the function is defined by *( ) ( )+ A function is convex if and only if its epigraph is a convex set and, equivalently, that a function is concave if and only if its hypograph is a convex set. 2.3 Differentiable Convex functions: Now we focus on differentiable convex and concave functions. A function an open convex set S of is said to be differentiable if the gradient defined on ( ). ( ) ( ) ( ) ( ) / exists at each First order differentiable condition: Let S be a nonempty open convex set in, and let be differentiable on S. is convex if and only if ( ) ( ) ( ) ( ) for all.

10 Second order differentiable condition: Let S be a nonempty set in, and let. Then, is said to be twice differentiable if there exist a vector ( ) and a symmetric matrix H( ), called the Hessian matrix, such that ( ) ( ) ( ) ( ) ( ) ( )( ). For twice differentiable function the Hessian matrix partial derivatives and is given by ( ) is given comprised of the second order ( ) [ ( ) ( ) ( ) ( ) ] (x) (x)... (x) (x) (x)... (x) (x) (x)... (x) Examples Let ( ) We have ( ) [ ] ( ) 0 1 Taking ( ) the second-order expansion of this function is given by ( ) ( ). / ( ) 0 1. / Since the given function is quadratic, no remainder term is present there, and so the above representation is exact. and so ( ) 0 1 ( ) 0 1 We conclude that is strictly concave. ( ) is positive definite and so ( ) is negative definite and the function

11 Examples Let ( ) We have ( ) [ ] ( ) [ ] Hence, the second-order expansion of this function about the point ( ) is given by ( ) ( ) ( ) ( ) 0 1 ( ) Theorem: Let S be a nonempty set in, and let be twice differentiable on S then, is convex on S if and only if the Hessian matrix is positive semi definite (PSD) at each point in S; that is, for any in S, we have ( ) Symmetrically, a function is concave on S if and only if its Hessian matrix is negative semi definite (NSD) everywhere in S, that is, for any, we have ( ) 2.4 Generalizations of a convex function: In this section, we present various types of functions that are similar to convex and concave functions but that share only some of their desirable properties. We ll learn that many of the results do not require the restrictive assumptions of convexity, but rather the less restrictive assumptions of quasi-convexity, pseudo-convexity Quasi-convex functions: A function defined on a convex subset S of a real vector space is quasi-convex if for any and, - furthermore if ( ( ) ) * ( ) ( )+ ( ( ) ) * ( ) ( )+ for any and, -, then is strictly quasi-convex. The function is said to be quasiconcave if is quasi-convex and a strictly quasi-concave functions a function whose negative is strictly quasi-convex. Equivalently a function is quasiconcave if ( ( ) ) * ( ) ( )+

12 and strictly quasi-concave if ( ( ) ) * ( ) ( )+ Examples of Quasi-convex functions: is quasi-convex on R. is quasi-linear on Differentiable Quasi-convex functions: Let S be a nonempty open convex set in, and let be differentiable on S. Then, is quasiconvex if and only if either one of the following equivalent statements holds: If ( ) ( ) ( ) ( ). If ( ) ( ) ( ) ( ). To illustrate the above theorem, let ( ). To check quasi-convexity, suppose ( ) ( ). This is true only if Now consider ( )( ) ( ). Since, ( ) therefore, ( ) ( ) ( )( ) and hence by the above theorem, is quasi-convex. Consider let ( ) Let ( ) and ( ). ( ) and ( ), so that ( ) ( ). But, ( ) ( ) ( )( ). By the necessary part of the theorem, is not quasiconvex. This shows that the sum of two quasi-convex functions is not necessarily quasiconvex. Propositions: A function which is both quasi-convex and quasi-concave is called quasi-linear. Every convex function is quasi-convex. A concave function can be quasi-convex function. For example log(x) is concave, and it is quasi-convex. If ( ) and ( ) are positive convex decreasing functions, then, ( ) ( ) is quasiconvex. Sum of two Quasi-convex functions is not necessarily a Quasi-convex.

13 2.4.3 Strongly Quasi-convex function: The concept of strong convexity extends the notion of strict convexity. Let S be a nonempty open convex set in, and let. The function is said to be strongly quasiconvex if for each we have ( ( ) ) * ( ) ( )+ for each ( ), The function is said to be strongly quasi-concave if is strongly quasi-convex. From the definition of convex function, quasi-convex function and strictly quasi-convex function, the following statements hold: Every strictly convex function is strongly Quasi-convex. Every strongly quasi-convex function is strictly quasi-convex. Every strongly quasi-convex function is quasi-convex Pseudo-convex functions: The differentiable strongly (or strictly) quasi-convex functions do not share the property of convex functions, which states that if ( ) at some then is a global minimum of. This motivates the definition of pseudo-convex functions which share this important property with convex functions, and leads to a generalization of various derivative based optimality conditions Definition Let S be a nonempty open convex set in, and let be differentiable on S. The function is said to be pseudo-convex if for each ( ) ( ) we have ( ) ( ); or, equivalently, if ( ) ( ) then ( ) ( ). The function is said to be pseudo-concave if is pseudo-convex. The function is said to be strictly pseudo-convex if, for each distinct satisfying ( ) ( ), we have ( ) ( ); or equivalently, if for each distinct ( ) ( ) implies that ( ) ( ). The function is said to be strictly pseudo-concave if is strictly pseudoconvex.

14 2.5 Relationship among various types of convex functions: Strictly convex Under differentiability Convex Strictly pseudo-convex Under differentiability Pseudo-convex Strongly quasi-convex Strictly quasi-convex Under lower semi continuity Quasi-convex

15 Chapter 3 Constrained Minimization: 3.1 Introduction and problem formulation: The majority of engineering problems involve constrained minimization- that is, the task is to minimize a function subject to constraints. Constrained problem may be expressed in the following general nonlinear programming form: Minimize ( ) subject to ( ) (3.1.1) and ( ) Where ( ) is a column vector of n real-valued design variables. In equation (3.1.1), is the objective or cost function, are inequality constraints and, are equality constraints. A vector satisfying all the constraints is called a feasible solution to the problem. The collection of all such solutions forms the feasible region. The nonlinear programming problem, then, is to find a feasible point such that ( ) ( ) for each feasible point x. Such a point is called an optimal solution. We define the feasible set Ω as Ω={ ( ) ( ) } The phrase minimization problem refers to finding a local minimizer in the feasible set Ω. 3.2 Graphical method: The graphical method for solving mathematical programming problem is based on a well-defined set of logical steps. Using graphical method, the given programming problem can be easily solved with a minimum amount of computational effort. We know that the simplex method is the well-studied and widely useful method for solving linear programming problem, while for the class of nonlinear programming no such widely useful method exists. Programming problems involving only two variables can easily solved graphically. The algorithms or the systematic procedure is discussed as follows: Algorithm: The solution nonlinear programming problem by graphical method, in general, involves the following steps: Step-1: Construct the graph of the given nonlinear programming problem. Step-2: Identify the convex region (solution space) generated by the objective function and constraints of the given problem.

16 Step-3: Determine the point in the convex region at which the objective function is optimum (maximum or minimum). Step-4: Interpret the optimum solution so obtained Solution of various kinds of problems by graphical method: Case -1: Problems with linear objective function and nonlinear constraints: Consider the problem Maximize Subject to (3.2.1) We ll solve this problem by graphical method. A B C O D Figure 3.1

17 For this, we have to trace the graph of the constraints of the problem considering inequalities as equations in the first quadrant (since ). We get the above convex region OABCD. We have to find a point which maximize the value of Z and also lies in the convex region OABCD. The desired point is obtained by moving parallel to for some k, so long as touches the extreme boundary point of the convex region OABCD. Hence C (2, 4) gives the maximum value of Z. at. Case-2 Problems with linear objective function and nonlinear as well as linear constraints: Consider the problem Maximize Subject to (3.2.2) We ll solve this problem by graphical method. A (0, 1) B. / O (0, 0) C (1, 0) Figure 3.2

18 For this we see that our objective function is linear and constraints are nonlinear. Constraints one is a circle of radius 1 with center at (0, 0) and constraints two is a straight line. OABC is the convex region having extreme points O (0, 0), A (0, 1), B. / and C (1, 0). Max at and. The nonlinear programming problems having more than two variables can t be solved using graphical method. In such cases we have to use the method of Lagrange multipliers. 3.3 Lagrange multipliers method: The basic ideas behind Lagrange multipliers can be illustrated by consider the following special case of the problem defined by Eq. (3.3.1) with three variables and only two equality constraints: Subject to Min ( ) ( ) (3.3.1) ( ) The feasible set Ω is a curve in, determined by the intersection of two surfaces given by and. We assume that ( ) is a point of minimum in Ω and that the functions have continuous first order derivatives on some open set containing and ( ), ( ) are linearly independent. Since ( ), are linearly independent and continuous, can be solved in term of as ( ) ( ) in the neighborhood around. Further u and v are differentiable around. Hence, Eq. (3.3.1) reduces to an unconstrained minimization problem of the following form ( ( ) ( )) (3.3.2) involving a single variable. Since is a point of minimum of Eq. (3.3.2), we have ( ) (3.3.3) holds at ( ). Also, from the constraint equations, we have two additional relations

19 (3.3.4) at ( ). From Eq. (3.3.3) and Eq. (3.3.4) [ ], ( ) ( ) ( )- Since (1, ) is not a null vector, we have det A= 0, which implies ( ) ( ) ( ) (3.3.5) where a, b, c are not all zero. But as ( ), ( ) are linearly independent and hence there exist, both not simultaneously zero, such that ( ) ( ) ( ) (3.3.6) Here are called Lagrange multipliers. So, in the equality constrained minimization problems, we have to find the critical point of the Lagrange function ( ) ( ) ( ) ( ) (3.3.7) instead of the critical points of ( ). The minimum of the Lagrangian function is also a minimum of, because equality constraints at the critical point imply that And hence ( ) ( ) ( ) ( ) The above analysis helps us in obtaining the following generalization of the above fact the n-dimensional minimization problem with equality constraints: min ( ) subject to (3.3.8) ( )

20 Example Solve the problem Min subject to Solution: The Lagrangian ( ) is given by ( ) ( ) The critical points of the Lagrangian ( ) are given by ( ) ( ) ( ) Solving (1) and (2) and substituting in (3) we get, ( ) ( ) ( ) ( ) ( ) The Hessian matrix of ( ) with respect to is given by ( ) 0 1 The Hessian matrix is positive definite at ( ) and negative definite at ( ) Hence ( ) is a minimizer and ( ) is a maximizer for the above problem. Example Minimize ( )= subject to Solution: The Lagrangian ( ) is given by

21 ( ) ( ) ( ) The critical points of the Lagrangian ( ) are given by ( ), we get Solving these five equations, we get and. The Hessian of ( ) with respect to is [ ], which is positive definite and hence ( ) is a point of minimum. Min. /. / Example Minimize Subject to We have ( ) The optimality conditions are Substituting for x (λ) into the constraint equation we get for each root, we obtain Point A in fig.3.1 Point B in fig.1

22 The minimum point is A. B( ). A( ) Figure 3.3

23 Chapter 4 Optimality conditions: 4.1 The Fritz John (FJ) optimality conditions: We now reduce the necessary optimality conditions to a statement in terms of gradients of the objective function and of the constraints. The resulting optimality conditions, credited to Fritz John, are given below. The Fritz John conditions are a necessary condition for a solution in nonlinear programming to be optimal. The concept of Kuhn-tucker conditions and Duality are central discussion on constrained optimization Theorem (The Fritz John (FJ) necessary conditions): Let X be a nonempty open set in and let, and for i=1,2,,m. Consider the problem in Eq to Minimize ( ) subject to ( ) (4.1.1) and ( ), Let be a feasible solution, and denote * ( ) + Suppose that for is continuous at, that are differentiable at, and that for is continuously differentiable at If locally solves above problems, then there exists scalars and for such that ( ) ( ) ( ) (4.1.2) ( ) Where is the vector whose components are and ( ). Furthermore, if each for i L I is also differentiable at, then the Fritz John conditions can be written in the following equivalent form

24 ( ) ( ) ( ) ( ) (4.1.3) ( ) In the condition of above theorem, the scalars for are usually called Langrage multipliers. The condition that is feasible to the problem Eq. (4.1.1) is called the primal feasibility (PF) condition, whereas the requirements ( ) ( ) ( ), and are sometimes called to as dual feasibility (DF) conditions. The condition ( ) for is called the complementary slackness (CS) condition. Together, the PF, DF, and the CS conditions are called the Fritz John (FJ) optimality conditions. Any point for which there exist Lagrangian multipliers ( ) such that ( ) satisfy the FJ conditions is called a Fritz John (FJ) point. Example Minimize ( ) ( ) Subject to The feasible region for the above problem is illustrated in figure 3.2. The Fritz -John conditions are true at the optimal point (2, 1). The set of binding constraints I at ( ) is given by I= {1, 2}. Thus, the Lagrangain multipliers and associated with and respectively, are equal to zero. ( ) ( ) ( ) ( ) ( ) ( )

25 (0, 2) ( ) ( ) (2, 1) ( ) (0, 0) ( ) Figure 4.1 Hence, to satisfy the Fritz John conditions, we need a nonzero vector ( ) satisfying ( ) ( ) ( ) ( ) This implies and. Taking and as such for any, we satisfy the Fritz Jhon conditions. Let us check whether the point ( ) is a FJ point, here, the set of binding constraints is * + and thus Note that ( ) ( ) ( ) ( ) ( ) ( ) Also the DF condition ( ) ( ) ( ) ( ) holds true if and only if and. If, then, contradicting the nonnegativity restrictions. If, on the other hand,, then, which contradicts the stipulation that the vector ( ) is nonzero. Thus, the Fritz John conditions do not hold true at ( ), which also shows that the origin is not a local optimal point.

26 4.2 KKT optimality conditions with both Equality and Inequality constraints: In the Fritz John conditions, the Langragian multiplier associated with the objective functions not necessarily positive. Under further assumptions on the constraint set we can show that at any local minimum, there exists a set of Langragian multiplier for which is positive. So we obtain a generalization of the KKT necessary optimality conditions KKT necessary optimality conditions: Let X be a nonempty open set in and let, and for i=1,2,,m. Consider the problem to Minimize ( ) subject to ( ) (4.2.1) and ( ), Let be a feasible solution, and denote * ( ) + Suppose that for is continuous at, that are differentiable at, and that for is continuously differentiable at We assume that is a regular point (gradients of active inequalities and of all the equality constraints are linearly independent). If locally solves above problems, and each for i L I is also differentiable at, then there exists scalars and for exist such that ( ) ( ) ( ) ( ) (4.2.2) The Lagrange multipliers associated with the equality constraints are unrestricted in sign they can be positive or negative; only the multipliers associated with the inequalities have to be nonnegative KKT Sufficient Conditions for Optimality: The KKT conditions are necessary conditions: if a point is local minimum to the problem in Eq. (4.1.1), then it should satisfy Eq. (4.2.2). Conversely, if a point satisfies Eq. (4.2.2), it can be a local minimum or a saddle point (neither a minimum nor a maximum). A set of sufficient

27 conditions will now be stated which, if satisfied, will ensure that the KKT point is a strict local minimum. Let be twice-continuously differentiable functions. Then, the point is a strict local minimum to Eq, (4.1.1) if there exit Lagrange multipliers µ and λ, such that KKT necessary conditions in Eq. (4.2.2) are satisfied at and The Hessian matrix ( ) ( ) ( ) ( ) (4.2.3) is positive definite on a subspace of as defined by the condition: ( ) for every vector which satisfies ( ) ( ) for all i for which ( ) (4.2.4) Consider the following two remarks to understand Eq. (4.2.4): (1) To understand what y is, consider a constraint h(x) = 0. This can be graphed as a surface in x-space. Then, ( ) is normal to the tangent plane to the surface at. Thus, if ( ) that means y is perpendicular to and is hence in the tangent plane. The collection of all such y s will define the entire tangent plane. This tangent plane is a subspace of the entire space. Thus, the KKT sufficient conditions only require ( ) to be positive definite on the tangent space M defined by M={ ( ) ( ) ( ) } (2) If ( ) for every vector then the Hessian matrix ( ) is positive definite (on the whole space ). Then it will also positive definite on a (tangent) subspace of. Example For the problem Minimize ( ) ( ) Subject to 2 (i) (ii) Write down the KKT conditions and find out the KKT points. Graph the problem and check whether the KKT point is a solution to the problem.

28 (iii) (iv) Sketch the feasible cone at the points (1, 0); that is, sketch the intersection of the descent and feasible cones. What is the feasible cone at the KKT point obtained in (i)? We have ( ) The KKT conditions ( ) ( ) ( ) Owing to the switching nature of the conditions we need to guess which constraints are active and then proceed with solving the KKT conditions. We have the following cases: (only 1 st constraints active) (1 st and 2 nd constraints active) Now, in case-2: We get, upon using the KKT conditions, Since is negative, we have violated the KKT conditions and here our guess is incorrect. The correct guess is Case 1:. This leads to: Substituting this into gives We can then recover Thus, we have the KKT point

29 The feasible cone at ( ) is shown in figure below. At the point ( ) we have the negative cost gradient to be ( ) ( ) and the active constraint gradient to be ( ). Since these two vectors are aligned with one another, there is no descent/feasible cone (or the cone is empty). (0,2) ( ) Feasible feasible cone at (1,0) region,ω Figure 4.2 (1,0) 4.3 Duality: Consider the general nonlinear programming problem Minimize ( ) subject to ( ) (4.3.1) and ( ) Where ( ) is a column vector of n real-valued design variables. In equation (4.3.1), is the objective or cost function,, are inequality constraints and, are equality constraints. First, we assume that all functions are twice-continuously differentiable and a solution of the primal problem in Eq. (4.3.1) is a regular point satisfying the KKT necessary conditions given in the previous section. In the sufficient conditions for a minimum to Eq. (4.3.1), we only required the Hessian of the Lagrangian ( ) be positive definite on the tangent

30 subspace. Now, to introduce duality, we have to make the stronger assumption that the Hessian of the Lagrangian ( ) is positive definite. This is a stronger assumption because we require positive definiteness of the Hessian at on the whole space and not just on a subspace. This is equivalent to requiring L to be locally convex at. We can state the following equivalent theorems, under these assumptions: (1) together with and, is solution to the primal problem of Eq.(4.3.1) (2) is a saddle point of the Lagrangian function ( ). That is, ( ) ( ) ( ) in the neighborhood of. (3) solves the dual problem Maximize ( ) ( ) ( ) (4.3.2) subject to ( ), ( )-, ( )- (4.3.3) and (4.3.4) where both x and Lagrange multipliers ( ) ( ) are variable and the two extrema are equal: From the above discussion the following two corollaries follows: (a) Any x that satisfies the constraints in Eq. (4.3.1) is called primal-feasible and any x, µ, λ that satisfies Eq. (4.3.3) and (4.3.4) is called dual-feasible. We know that any primalfeasible point provides us with a upper bound on the optimum cost: ( ) ( ) Any dual feasible point provides a lower bound on the optimum cost as ( ) ( ) It is important to realize that the lower bound is only on the local minimum of f in the neighborhood. (b) If the dual objective is unbounded, then the primal has no feasible solution. 4.4 The Dual Function ( ): The conditions in Eq. (4.3.3), i.e. ( ) ( ) ( ), can be viewed as equations involving two sets of variables: x and [µ, λ]. The Jacobian with respect to the x variables of this system, is. Since this matrix is non-singular at (by the hypothesis that it is

31 positive definite), we can invoke the Implicit Function Theorem which states that in a small neighborhood of, -, such that for µ, λ in this neighborhood, ( ) is a differentiable function of with ( ( ) ) This gives us to treat L as an implicit function of as ( ( ) ) which is called the dual function ( ). Since will remain positive definite near, we can define ( ) from the minimization problem ( ) ( ) ( ) ( ) (4.4.1) Where are near and where the minimization is carried out with respect to x near Gradient and Hessian of ( ) in Dual Space: The gradient of the dual function is, ( )- (4.4.2), ( )- (4.4.3) The proof follows from: ( ( ) ) in view of the fact that by the very construction of. The Hessian of in space is given by, ( )- Where the rows of A are the gradient vectors of the constraints. Since we assume that regular point, A has full row rank and hence the dual Hessian is negative definite. is a Example: Minimize ( ) ( ) Subject to We have ( ) ( ) ( ). It is clear that is positive definite. Setting we obtain

32 Substituting into L, we have ( ) From Eq. (4.4.2) and eq. (4.4.3) we have we have The dual problem is max ( ) with the restriction that The solution is Correspondingly, we obtain ( ) and ( ).

33 Chapter -5 References: 1) Nonlinear programming theory and algorithms, second edition, Mokhtar S. Bazaraa, Hanif D. Sherali, C.M. Shetty. John Wiley and sons publication 2) Optimization concepts and applications in engineering, Ashok D. Belegundu, Tirupathi R. Chandrupatla. Published by Pearson education (Singapore) Pte. Ltd. 3) Optimization theory and practice, Mohan C Joshi, Kannan M Moudgalya Narosa Publishing House, ) Effect of graphical method for solving mathematical programming problem, Bimal Chandra Das, Department of Textile Engineering, Daffodil International University 28 journal of Science and Technology, Vol. 4, January 2009, Dhaka, Bangladesh 5) Convex optimization, Stephen Boyd and Lieven Vandenerghe, Cambridge University Press (2009) 6) Principles of Optimization Theory, C.R. Bector, S. Chandra, J. Dutta Narosa Publishing House 7) Convexity and optimization in finite dimension I, Josef Stoer, Christoph Witzgall Springer Verlag Berlin.Heidelberg.New york