Lecture 3: Basics of set-constrained and unconstrained optimization (Chap 6 from textbook) Xiaoqun Zhang Shanghai Jiao Tong University Last updated: October 9, 2018
Optimization basics Outline Optimization basics
Optimization basics General form of optimization Ω is called the feasible set. min f (x) s.t x Ω x Rn If Ω = R n, the problem is called unconstrained; otherwise, it is constrained. If we allow f to have the extended value, then we can write any constrained problem in the unconstrained form min f (x) + ι Ω (x) where the indicator function ι Ω (x) = the objective function f (x) + ι Ω (x) is nonsmooth. { 0, x Ω;, x / Ω.
Optimization basics Minimizers Suppose that f : R n R is a real-valued function on Ω R n. A point x Ω is a local minimizer of f over Ω if there exists ɛ > 0 such that f (x) f (x ) for all x Ω \ x and x x < ɛ. A point x Ω is a global minimizer of f over Ω if f (x) f (x ) for all x Ω \ {x }.
Optimization basics Feasible direction A vector d R n is a feasible direction at x Ω at x Ω if d 0 and x + αd Ω for some mall α > 0. If Ω = R n or x lies in the interior of Ω, then any d R n /{0} is a feasible direction. Feasible directions are introduced to establish optimality conditions, especially for points on the boundary of a constrained problem.
Optimization basics Example: linear equality Equality constraints: hyperplane {x R n : u T x = v} where u = [u 1,, u n ] T. For a given feasible point x, any direction d satisfying u T d = 0(d 0) is a feasible direction at x.
Optimization basics Example: linear inequality Inequality constraints: half space {x R n : u T x v}. Given a feasible point x. If the constraint is active at x, i.e. u T x = v, then any direction d satisfying u T d 0(d 0) is a feasible direction at x ; If the constraint is inactive at x, i.e. u T x < v, then any direction d 0 is a feasible direction at x.
Optimization basics Directional derivatives min f (x) x R n,x Ω Let f : R n R be a real valued function and let d be a feasible direction at x Ω. The directional derivative of f in the direction d, denoted as Suppose that x and d are given, we have If d = 1, then f d f d f (x + αd) f (x) lim α 0 α f d = d dα f (x + αd) α=0 = f (x) T d is the rate of increase of f at x in the direction d.
Optimization basics Example Define f : R 3 R by f (x) = x 1 x 2 x 3 and let d = [ 1 2, 1 2, 2 1 ] T. The directional derivative of f in the direction d is 1 f d = f 2 (x)t d = [x 2 x 3, x 1 x 3, x 1 x 2 ] 1 2 = x 2x 3 + x 1 x 3 + 2x 1 x 2 2 as d = 1, the above is also the rate of increase of f at x in the direction d. 1 2
Outline Optimization basics
First Order Necessary Condition(FONC) Theorem Let Ω be a subset of R n and f C 1 a real valued function on Ω, if x is a local minimizer of f over Ω. Then for any feasible direction d at x, we have d T f (x ) 0 Proof: Define x(α) = x + αd Ω for any feasible direction d and α 0. Define φ(α) = f (x + αd) and by Taylor s Theorem: f (x + αd) f (x ) = φ(α) φ(0) = αφ (0) + o(α) = αd T f (x ) + o(α) Thus if d T f (x ) < 0, then for sufficient small α, f (x + αd) < f (x ) which is contradiction with the fact that x is a local minimizer.
Geometry illustration
FONC for Interior case Corollary Let Ω be a subset of R n and f C 1 a real valued function on Ω, if x is a local minimizer of f over Ω and if x is an interior point of Ω, then f (x ) = 0. Proof: As any d R n \ 0 is a feasible direction, we can set d = f (x ). From the above theorem, we have d T f (x ) = f (x ) T f (x ) = f (x ) 2 0. As f (x ) 2 0, we must have f (x ) 2 = 0 and thus f (x ) = 0. Remarks: Interior case also reduces to the problem min f (x) to solve f (x ) = 0. FONC: f d (x ) 0 for all feasible directions d. i.e. the rate of increase of f at x in any feasible direction d in Ω is nonnegative.
Example Consider the problem min x 2 1 + 1 2 x2 2 + 3x 2 + 9 2 s.t.x 1, x 2 0 Is the FONC for a local minimizer satisfied at (a) x = [1, 3] T ; (b) x = [0, 3] T ; (c) x = [1, 0] T ; (d) x = [0, 0] T Use Matlab command ( contour3 ):
Solution 1 At x = [1, 3] T, f (x) = [2, 6] T. The point is an interior point. Hence FONC requires f (x) = 0, which is not. Thus x = [1, 3] T is not a local minimizer. 2 At x = [0, 3] T, f (x) = [0, 6] T. And f (x) T d = 6d 2 where d = [d 1, d 2 ]. For d to be feasible, we need d 1 0 and d 2 can be arbitrary. If we let d = [1, 1] T, then d T f (x) = 6 < 0, then x = [0, 3] T is not a local minimizer. 3 At [1, 0] T, f (x) = [2, 3] T, For d = [d 1, d 2 ] T to be feasible, d 1 arbitrary, d 2 0 and d T f (x) = 2d 1 + 3d 2. If we take d = [ 2, 1] T, d T f (x) = 1 < 0, thus x = [1, 0] T is not a local minimizer. 4 At x = [0, 0] T, f (x) = [0, 3] T. Hence d T f (x) = 3d 2 where d = [d 1, d 2 ]. For d to be feasible, d 1 0, d 2 0. hence d T f (x) 0. Hence x = [0, 0] T satisfy the FONC for a local minimizer. In fact, it is global minimizer.
Second order necessary condition (SONC) Let C 2 be the twice continuously differentiable functions space. Theorem Let Ω be a subset of R n and f C 2 be a real valued function on Ω. x is a local minimizer of f over Ω and d a feasible direction at d T f (x ) = 0, then where F is the Hessian of f. d T F(x )d 0 Proof: Assume that there exists a feasible direction d with d T f (x) = 0 and d T F(x )d < 0. By second order Taylor s expansion: f (x + αd) = f (x ) + α 2 dt F(x )d + o(α 2 ) 2 For sufficient small α > 0, we have f (x + αd) < f (x ) which contradicts that x is a local minimizer.
SONC for interior case Corollary Let x be an interior point of Ω R n. If x is a local minimizer of f : Ω R, f C 2, then f (x ) = 0 and F(x ) is a positive semidefinite (F(x ) 0); that is, for all d R n, d T F(x )d 0
The necessary conditions are not sufficient Counter examples f (x 1, x 2 ) = x1 2 x2 2. At x = 0,, f (x) = 0 while 0 is neither a local minimizer nor a local f (x) = x 3, f (x) = 3x 2, f (x) = 6x. At x = maximizer (saddle point). 0, f (x) = 0, while 0 is not a local minimizer.
Second-order sufficient condition (SOSC) Let f C 2 be defined on a region in which x is an interior point. Suppose that f (x ) = 0, F(x ) > 0 (F(x ) is positive definite). Then x is a strict local minimizer of f. The condition is not necessary for strict local minimizer. Proof: For any d 0 and d = 1, we have d T F(x )d λ min (F(x )) > 0 for x 0. Use the second order Taylor expansion f (x + αd) = f (x ) + α2 2 dt F(x )d + 0(α 2 ) f (x ) + α2 2 λ min(f(x )) + o(α 2 ) Then α > 0, regardless of d, such that f (x + αd) > f (x ), α (0, α).
Examples f (x) = x 2 1 + x 2 2, the point 0 satisfies the SOSC. For any unconstrained quadratic problem: min f (x) = 1 2 xt Qx b T x, if x satisfies the second order necessary condition, then x is a global minimizer. For any unconstrained quadratic problem with Q 0, x is a global minimizer if and only if x satisfies the first order necessary condition. That is, the problem is equivalent to solve Qx = b. Consider minimize c T x, subject to x Ω. Suppose that c 0 and the problem has a global minimizer. Can the minimizer lie in the interior of Ω?
Remarks on roles of optimality conditions Recognize a solution: given a candidate solution, check optimality conditions to verify if it is a solution Measure the quality of an approximate solution: stopping criteria Develop algorithms.