Introduction to Optimization Techniques Nonlinear Optimization in Function Spaces
X : T : Gateaux and Fréchet Differentials Gateaux and Fréchet Differentials a vector space, Y : a normed space transformation (possibly nonlinear) mapping D( X) R( Y) Definition Let x D X and let h be arbitrary in X. If the limit T( x; h) lim [ T( xh) T( x)] exists, it is called the Gateaux differential of T at x with increment h. If the limit exists for each h X, the transformation T is said to be Gateaux differentiable at x. The Gateaux differential generalizes the concept of directional derivative familiar in finite-dimensional space Fréchet differential: more satisfactory definition 2
Gateaux and Fréchet Differentials Definition Let T be a transformation defined on an open domain D in a normed space X and having range in a normed space Y. If for fixed x D and each h X there exists T( x; h) Y which is linear and continuous w.r.t. h such that T( xh) T( x) T( x; h) lim h h then T is said to be Fréchet differentiable at x and T( x; h) is said to be the Fréchet differential of T at x with increment h. 3
Gateaux and Fréchet Differentials Proposition. If the transformation T has a Fréchet differential, it is unique. Proposition 2. If the Fréchet differential of T exists at x, then the Gateaux differential exists at x and they are equal. Proposition 3. If the transformation T defined on an open set D in has a Fréchet differential at x, then T is continuous at x. X 4
Local Theory of Constrained Optimization Lagrange Multiplier Theorems Inverse Function Theorem T Definition Let be a continuously Fréchet differentiable transformation from an open set in a Banach space X into a Banach space Y. If x is such that maps onto, the point D T( x ) X Y is said to be a regular point of the transformation. T x T Ex. If is a mapping from into, a point is a regular point if the Jacobian matrix of has rank. E n T x m m E E n 5
Local Theory of Constrained Optimization Theorem. (Generalized Inverse Function Theorem) Let x be a regular point of a transformation T mapping the Banach space X into the Banach space Y. Then there is a neighborhood N( y) of the point y T( x) (i.e., a sphere centered at y ) and a constant K such that the equation T( x) has a solution for every and the solution satisfies y N( y ) x x K y y y 6
Local Theory of Constrained Optimization Equality Constraints f Gx ( ) Necessary conditions for an extremum of subject to where f is a real-valued functional on a Banach space X and G is a mapping from X into a Banach space Z. f Gx ( ) Lemma Let achieve a local extremum subject to at the point x and assume that f and G are continuously Fréchet differentiable in an open set containing x and that is x a regular point of. Then for all satisfying G( x ) h. G f( x ) h h 7
Local Theory of Constrained Optimization Proof To be specific, assume that the local extremum is a local minimum. Consider the transformation T : X RZ defined by T( x) ( f ( x), G( x)). If there were an h such that G( x, ) h f( x, then ) h T( x) ( f( x), G( x)): X RZ would be onto R Z since G( x is onto. By the inverse ) Z function theorem, it would follow that for any, there exists a vector x and with x x such that T( x) ( f( x ), ), contradicting the assumption that x is a local minimum. 8
Tangent Space Tangent space of the constraint surface. Tangent space at : N ( ( )) x G x { hz G( x ) h} Tangent space of the surface M { x: G( x) } near. x f is stationary at x with respect to variation in the tangent plane. f : const tangent space gx ( ) G( x ) x f ( x ) Constrained optimization 9
Lagrange Multiplier Theorem Defintion Let X be a normed linear vector space. The space of all bounded linear functionals on X is called the normed dual of X and is denoted X. The norm of an element f X is f sup f( x). x The value of a linear functional x X at the point x X is denoted by x ( x ) or by the more symmetric notation x, x. Theorem X is a Banach space.
Lagrange Multiplier Theorem Theorem (Lagrange Multiplier) If the continuously Fréchet differentiable functional f has a local extremum under the constraint Gx ( ) at the regular point x, then there exists an element z Z such that the Lagrangian functional Lxz (, ) f( x) Gx ( ), z, or Lxz (, ) f( x) zgx ( ) x stationary at, i.e., f( x ) z G( x ).
Lagrange Multiplier Theorem Proof. From Lemma it is clear that f( x ) is orthogonal to the nullspace of G( x. Since, however, the range of is closed, it ) G( x ) follows that f x R G x ( ) [ ( ) ] Theorem Let X and Y be normed spaces and let A BXY (, ). Then R N [ ( A )] ( A ) R [ ( )] ( ) A N A N N R ( A ) ( A ) R ( A ) ( A ) 2
Lagrange Multiplier Theorem Hence there is a z Z such that f( x ) G ( x ) z or f( x ) z G( x ) When x is not regular. Corollary. Assuming all the hypothesis of Theorem with the exception that the range of G( x ) is closed but perhaps not onto, there exists a nonzero element ( r such that, z) R Z the functional rf( x) zgx ( ) is stationary at x. 3
Lagrange Multiplier Theorem Ex. G consists of two functionals g, g2. For optimality the gradient of must lie in the plane generated by and g ; 2 hence f g f( x ) z g( x ) z g( x ) 2 2 g f g 2 g g 2 4
Inequality Constraints (Karush-Kuhn-Tucker Theorem) Derivation of the local necessary conditions minimize subject to f ( x) Gx ( ) f : X R G: X Z normed space with positive cone P 5
Inequality Constraints (Karush-Kuhn-Tucker Theorem) Ex. Consider a problem in two dimensions with three scalar equations g ( x) as constraints. i g ( x) 2 g ( x) g ( x) 2 x g ( x) g ( x) 3 g ( x) 3 (a) (b) g ( x) 2 x g f g 2 g x g g ( x) g ( x) ( ) f 2 x g ( x) g ( x) 3 3 (c) (d) 6
Inequality Constraints (Karush-Kuhn-Tucker Theorem) (b) x : interior of the region f x (c) The minimum occurs on the boundary g x. f x must be orthogonal to the boundary and point inside. f x g x for some (d) The minimum point x satisfies both and g x g 2 x f x g x g x with, 2 2 2 7
Inequality Constraints (Karush-Kuhn-Tucker Theorem) General Statement f x G x and, i, 2, 3 igi x If i g x i 8
Inequality Constraints (Karush-Kuhn-Tucker Theorem) Definition: Let X be a vector space and let Z be a normed space with a positive cone P having nonempty interior. Let G be a mapping G: X Z which has a Gateaux differential that is linear in its increment. A point x X is said to be a regular point of the inequality G x G x h if G x and there is an h X such that G x ; 9
Inequality Constraints (Karush-Kuhn-Tucker Theorem) Theorem. Let X be a space vector and Z a normed space having positive cone P. Assume that P contains an interior point. Let f be a Gateaux differentiable real-valued functional on X and G a Gateaux differentiable mapping from X into Z. Assume that the Gateaux differentials are linear in their increments. x Suppose minimizes f subject to G x and that x is a regular point of the inequality G x. Then there is a z Z, z such that the Lagrangian, f x G x z 2
Inequality Constraints (Karush-Kuhn-Tucker Theorem) x is stationary at ; furthermore G x, z. 2