Optimality Conditions for Unconstrained Problems

Transcription

1 62

2 CHAPTER 6 Opimaliy Condiions for Unconsrained Problems 1 Unconsrained Opimizaion 11 Exisence Consider he problem of minimizing he funcion f : R n R where f is coninuous on all of R n : P min f(x) x R n As we have seen, here is no guaranee ha f has a minimum value, or if i does, i may no be aained To clarify his siuaion, we examine condiions under which a soluion is guaraneed o exis Recall ha we already have a our disposal a rudimenary exisence resul for consrained problems This is he Weiersrass Exreme Value Theorem Theorem 11 (Weiersrass Exreme Value Theorem) Every coninuous funcion on a compac se aains is exreme values on ha se We now build a basic exisence resul for unconsrained problems based on his heorem For his we make use of he noion of a coercive funcion Definiion 11 A funcion f : R n R is said o be coercive if for every sequence {x ν } R n for which x ν i mus be he case ha f(x ν ) + as well Coninuous coercive funcions can be characerized by an underlying compacness propery on heir lower level ses Theorem 12 (Coerciviy and Compacness) Le f : R n R be coninuous on all of R n The funcion f is coercive if and only if for every α R he se {x f(x) α} is compac Proof We firs show ha he coerciviy of f implies he compacness of he ses {x f(x) α} We begin by noing ha he coninuiy of f implies he closedness of he ses {x f(x) α} Thus, i remains only o show ha any se of he form {x f(x) α} is bounded We show his by conradicion Suppose o he conrary ha here is an α R n such ha he se S = {x f(x) α} is unbounded Then here mus exis a sequence {x ν } S wih x ν Bu hen, by he coerciviy of f, we mus also have f(x ν ) This conradics he fac ha f(x ν ) α for all ν = 1, 2, Therefore he se S mus be bounded Le us now assume ha each of he ses {x f(x) α} is bounded and le {x ν } R n be such ha x ν Le us suppose ha here exiss a subsequence of he inegers J N such ha he se {f(x ν )} J is bounded above Then here exiss α R n such ha {x ν } J {x f(x) α} Bu his canno be he case since each of he ses {x f(x) α} is bounded while every subsequence of he sequence {x ν } is unbounded by definiion Therefore, he se {f(x ν )} J canno be bounded, and so he sequence {f(x ν )} conains no bounded subsequence, ie f(x ν ) This resul in conjuncion wih Weiersrass s Theorem immediaely yields he following exisence resul for he problem P Theorem 13 (Coerciviy implies exisence) Le f : R n R be coninuous on all of R n If f is coercive, hen f has a leas one global minimizer Proof Le α R be chosen so ha he se S = {x f(x) α} is non-empy By coerciviy, his se is compac By Weiersrass s Theorem, he problem min {f(x) x S } has a leas one global soluion Obviously, he se of global soluions o he problem min {f(x) x S } is a global soluion o P which proves he resul Remark 11 I should be noed ha we only need o know ha he coerciviy hypohesis is sronger han is sricly required in order o esablish he exisence of a soluion Indeed, a global minimizer mus exis if here exis 63

3 64 6 OPTIMALITY CONDITIONS FOR UNCONSTRAINED PROBLEMS one non-empy compac lower level se We do no need all of hem o be compac However, in pracice, coerciviy is easy o check 12 Firs-Order Opimaliy Condiions This exisence resul can be quie useful, bu unforunaely i does no give us a consrucive es for opimaliy Tha is, we may know a soluion exiss, bu we sill do no have a mehod for deermining wheher any given poin may or may no be a soluion We now presen such a es using he derivaives of he objecive funcion f For his we will assume ha f is wice coninuously differeniable on R n and develop consrucible firs- and second-order necessary and sufficien condiions for opimaliy The opimaliy condiions we consider are buil up from hose developed in firs erm calculus for funcions mapping from R o R The reducion o he one dimensional case comes abou by considering he funcions φ : R R given by φ() = f(x + d) for some choice of x and d in R n The key variaional objec in his conex is he direcional derivaive of f a a poin x in he direcion d given by f f(x + d) f(x) (x; d) = lim 0 When f is differeniable a he poin x R n, hen f (x; d) = f(x) T d = φ (0) Noe ha if f (x; d) < 0, hen here mus be a > 0 such ha In his case, we mus have f(x + d) f(x) < 0 whenever 0 < < f(x + d) < f(x) whenever 0 < < Tha is, we can always reduce he funcion value a x by moving in he direcion d an arbirarily small amoun In paricular, if here is a direcion d such ha f (x; d) exiss wih f (x; d) < 0, hen x canno be a local soluion o he problem min x R n f(x) Or equivalenly, if x is a local o he problem min x R n f(x), hen f (x; d) 0 whenever f (x; d) exiss We sae his elemenary resul in he following lemma Lemma 11 (Basic Firs-Order Opimaliy Resul) Le f : R n R and le x R n be a local soluion o he problem min x R n f(x) Then f (x; d) 0 for every direcion d R n for which f (x; d) exiss We now apply his resul o he case in which f : R n R is differeniable Theorem 14 Le f : R n R be differeniable a a poin x R n If x is a local minimum of f, hen f(x) = 0 Proof By Lemma 11 we have Taking d = f(x) we ge Therefore, f(x) = 0 0 f (x; d) = f(x) T d for all d R n 0 f(x) T f(x) = f(x) 2 0 When f : R n R is differeniable, any poin x R n saisfying f(x) = 0 is said o be a saionary (or, equivalenly, a criical) poin of f In our nex resul we link he noions of coerciviy and saionariy Theorem 15 Le f : R n R be differeniable on all of R n If f is coercive, hen f has a leas one global minimizer hese global minimizers can be found from among he se of criical poins of f Proof Since differeniabiliy implies coninuiy, we already know ha f has a leas one global minimizer Differeniabily implies ha his global minimizer is criical This resul indicaes ha one way o find a global minimizer of a coercive differeniable funcion is o firs find all criical poins and hen from among hese deermine hose yielding he smalles funcion value

4 1 UNCONSTRAINED OPTIMIZATION Second-Order Opimaliy Condiions To obain second-order condiions for opimaliy we mus firs recall a few properies of he Hessian marix 2 f(x) The calculus ells us ha if f is wice coninuously differeniable a a poin x R n, hen he hessian 2 f(x) is a symmeric marix Symmeric marices are orhogonally diagonalizable Tha is, here exiss and orhonormal basis of eigenvecors of 2 f(x), v 1, v 2,, v n R n such ha λ λ f(x) = V V T 0 0 λ n where λ 1, λ 2,, λ n are he eigenvalues of 2 f(x) and V is he marix whose columns are given by heir corresponding vecors v 1, v 2,, v n : V = [ v 1, v 2,, v n] I can be shown ha 2 f(x) is posiive semi-definie if and only if λ i 0, i = 1, 2,, n, and i is posiive definie if and only if λ i > 0, i = 1, 2,, n Thus, in paricular, if 2 f(x) is posiive definie, hen d T 2 f(x)d λ min d 2 for all d R n, where λ min is he smalles eigenvalue of 2 f(x) We now give our main resul on second-order necessary and sufficien condiions for opimaliy in he problem min x R n f(x) The key ools in he proof are he noions of posiive semi-definieness and definieness along wih he second-order Taylor series expansion for f a a given poin x R n : (67) f(x) = f(x) + f(x) T (x x) (x x)t 2 f(x)(x x) + o( x x 2 ) where (68) (69) o( x x 2 ) lim x x x x 2 = 0 Theorem 16 Le f : R n R be wice coninuously differeniable a he poin x R n (1) (Necessiy) If x is a local minimum of f, hen f(x) = 0 and 2 f(x) is posiive semi-definie (2) (Sufficiency) If f(x) = 0 and 2 f(x) is posiive definie, hen here is an α > 0 such ha f(x) f(x) + α x x 2 for all x near x Proof (1) We make use of he second-order Taylor series expansion (67) and he fac ha f(x) = 0 by Theorem 14 Given d R n and > 0 se x := x + d, plugging his ino (67) we find ha f(x + d) f(x) 0 = 1 2 dt 2 f(x)d + o(2 ) since f(x) = 0 by Theorem 14 Taking he limi as 0 we ge ha 0 d T 2 f(x)d Since d was chosen arbirarily, 2 f(x) is posiive semi-definie (2) The Taylor expansion (67) and he hypohesis ha f(x) = 0 imply ha f(x) f(x) x x 2 = 1 (x x) T (x x) 2 x x 2 f(x) x x + o( x x 2 ) x x 2 If λ min > 0 is he smalles eigenvalue of 2 f(x), choose ɛ > 0 so ha o( x x 2 ) x x 2 λ min 4 whenever x x < ɛ Then, for all x x < ɛ, we have from (68) and (69) ha f(x) f(x) x x λ min + o( x x 2 ) x x λ min Consequenly, if we se α = 1 4 λ min, hen f(x) f(x) + α x x 2

5 66 6 OPTIMALITY CONDITIONS FOR UNCONSTRAINED PROBLEMS whenever x x < ɛ In order o apply he second-order sufficien condiion one mus be able o check ha a symmeric marix is posiive definie As we have seen, his can be done by compuing he eigenvalues of he marix and checking ha hey are all posiive Bu here is anoher approach ha is ofen easier o implemen using he principal minors of he marix Theorem 17 Le H R n n be symmeric We define he kh principal minor of H, denoed k (H), o be he deerminan of he upper-lef k k submarix of H Then (1) H is posiive definie if and only if k (H) > 0, k = 1, 2,, n (2) H is negaive definie if and only if ( 1) k k (H) > 0, k = 1, 2,, n Definiion 12 Le f : R n R be coninuously differeniable a x If f(x) = 0, bu x is neiher a local maximum or a local minimum, we call x a saddle poin for f Theorem 18 Le f : R n R be wice coninuously differeniable a x If f(x) = 0 and 2 f(x) has boh posiive and negaive eigenvalues, hen x is a saddle poin of f Theorem 19 Le H R n n be symmeric If H is nieher posiive definie or negaive definie and all of is principal minors are non-zero, hen H has boh posiive and negaive eigenvalues In his case we say ha H is indefinie Example 11 Consider he marix H = We have 1 (H) = 1, 2 (H) = = 4, and 3(H) = de(h) = 8 Therefore, H is posiive definie 14 Convexiy In he previous secion we esablished firs- and second-order opimaliy condiions These condiions we based on only local informaion and so only refer o properies of local exrema In his secion we sudy he noion of convexiy which allows us o provide opimaliy condiions for global soluions Definiion 13 (1) A se C R n is said o be convex if for every x, y C and λ [0, 1] one has (1 λ)x + λy C (2) A funcion f : R n R is said o be convex if for every wo poins x 1, x 2 R n and λ [0, 1] we have (70) f(λx 1 + (1 λ)x 2 ) λf(x 1 ) + (1 λ)f(x 2 ) The funcion f is said o be sricly convex if for every wo disinc poins x 1, x 2 R n and λ (0, 1) we have (71) f(λx 1 + (1 λ)x 2 ) < λf(x 1 ) + (1 λ)f(x 2 ) The inequaliy (70) is equivalen o he saemen ha he secan line connecing (x 1, f(x 1 )) and (x 2, f(x 2 )) lies above he graph of f on he line segmen λx 1 + (1 λ)x 2, λ [0, 1] (x 1, f (x 1)) (x, f (x )) 2 2 x 1 λx 1 + (1 - λ)x 2 x 2

6 1 UNCONSTRAINED OPTIMIZATION 67 Tha is, he se epi (f) = {(x, µ) : f(x) µ}, called he epi-graph of f is a convex se Indeed, i can be shown ha he convexiy of he se epi (f) is equivalen o he convexiy of he funcion f This observaion allows us o exend he definiion of he convexiy of a funcion o funcions aking poenially infinie values Definiion 14 A funcion f : R n R {+ } = R is said o be convex if he se epi (f) = {(x, µ) : f(x) µ} is a convex se We also define he essenial domain of f o be he se dom (f) = {x : f(x) < + } We say ha f is sricly convex if he sric inequaliy (71) holds whenever x 1, x 2 dom (f) are disinc Example 12 c T x, x, e x, x 2 The role of convexiy in linking he global and he local in opimizaion heory is illusraed by he following resul Theorem 110 Le f : R n R be convex If x R n is a local minimum for f, hen x is a global minimum for f Proof Suppose o he conrary ha here is a x R n wih f( x) < f(x) Since x is a local soluion, here is an ɛ > 0 such ha f(x) f(x) whenever x x ɛ Taking ɛ smaller if necessary, we may assume ha ɛ < 2 x x Se λ := ɛ(2 x x ) 1 < 1 and x λ := x + λ( x x) Then x λ x ɛ/2 and f(x λ ) (1 λ)f(x) + λf( x) < f(x) This conradics he choice of ɛ and so no such x exiss Sric convexiy implies he uniqueness of soluions Theorem 111 Le f : R n R be sricly convex If f has a global minimizer, hen i is unique Proof Le x 1 and x 2 be disinc global minimizers of f Then, for λ (0, 1), f((1 λ)x 1 + λx 2 ) < (1 λ)f(x 1 ) + λf(x 2 ) = f(x 1 ), which conradics he assumpion ha x 1 is a global minimizer (72) If f is a differeniable convex funcion, much more can be said We begin wih he following lemma Lemma 12 Le f : R n R be convex (no necessarily differeniable) (1) Given x, d R n he difference quoien f(x + d) f(x) is a non-decreasing funcion of on (0, + ) (2) For every x, d R n he direcional derivaive f (x; d) always exiss and is given by (73) f (x; d) := inf >0 f(x + d) f(x) Proof We firs assume (1) is rue and show (2) Recall ha (74) f f(x + d) f(x) (x; d) := lim 0 Now if he difference quoien (72) is non-decreasing in on (0, + ), hen he limi in (74) is necessarily given by he infimum in (73) This infimum always exiss and so f (x; d) always exiss and is given by (73) We now prove (1) Le x, d R n and le 0 < 1 < Then f(x + 1 d) = ( f x + [( = f 1 ( 1 ) ) d ( 1 )) x + ( ) 1 1 f(x) + ( ) ] 1 (x + d) ( 1 ) f(x + d)

7 68 6 OPTIMALITY CONDITIONS FOR UNCONSTRAINED PROBLEMS Hence f(x + 1 d) f(x) 1 f(x + d) f(x) A very imporan consequence of Lemma 12 is he subdifferenial inequaliy This inequaliy is obained by plugging = 1 and d = y x ino he righ hand side of (73) where y is any oher poin in R n This subsiuion gives he inequaliy (75) f(y) f(x) + f (x; y x) for all y R n and x dom f The subdifferenial inequaliy immediaely yields he following resul Theorem 112 (Convexiy and Opimaliy) Le f : R n R be convex (no necessarily differeniable) and le x dom f Then he following hree saemens are equivalen (i) x is a local soluion o min x R n f(x) (ii) f (x; d) 0 for all d R n (iii) x is a global soluion o min x R n f(x) Proof Lemma 11 gives he implicaion (i) (ii) To see he implicaion (ii) (iii) we use he subdifferenial inequaliy and he fac ha f (x; y x) exiss for all y R n o obain f(y) f(x) + f (x; y x) f(x) for all y R n The implicaion (iii) (i) is obvious If i is furher assumed ha f is differeniable, hen we obain he following elemenary consequence of Theorem 112 Theorem 113 Le f : R n R be convex and suppose ha x R n is a poin a which f is differeniable Then x is a global minimum of f if and only if f(x) = 0 As Theorems 112 and 113 demonsrae, convex funcions are well suied o opimizaion heory Thus, i is imporan ha we be able o recognize when a funcion is convex For his reason we give he following resul Theorem 114 Le f : R n R (1) If f is differeniable on R n, hen he following saemens are equivalen: (a) f is convex, (b) f(y) f(x) + f(x) T (y x) for all x, y R n (c) ( f(x) f(y)) T (x y) 0 for all x, y R n (2) If f is wice differeniable hen f is convex if and only if 2 f(x) is posiive semi-definie for all x R n Remark 12 The condiion in Par (c) is called monooniciy Proof (a) (b) (b) (c) and If f is convex, hen 114 holds By seing := 1 and d := y x we obain (b) Le x, y R n From (b) we have f(y) f(x) + f(x) T (y x) f(x) f(y) + f(y) T (x y) By adding hese wo inequaliies we obain (c) (c) (b) Le x, y R n By he Mean Value Theorem here exiss 0 < λ < 1 such ha where x λ := λy + (1 λ)x By hypohesis, Hence f(y) f(x) + f(x) T (y x) f(y) f(x) = f(x λ ) T (y x) 0 [ f(x λ ) f(x)] T (x λ x) = λ[ f(x λ ) f(x)] T (y x) = λ[f(y) f(x) f(x) T (y x)]

8 1 UNCONSTRAINED OPTIMIZATION 69 (b) (a) Le x, y R n and se α := max ϕ(λ) := [f(λy + (1 λ)x) (λf(y) + (1 λ)f(x))] λ [0,1] We need o show ha α 0 Since [0, 1] is compac and ϕ is coninuous, here is a λ [0, 1] such ha ϕ(λ) = α If λ equals zero or one, we are done Hence we may as well assume ha 0 < λ < 1 in which case where x λ = x + λ(y x), or equivalenly Bu hen by (b) 0 = ϕ (λ) = f(x λ ) T (y x) + f(x) f(y) λf(y) = λf(x) f(x λ ) T (x x λ ) α = f(x λ ) (f(x) + λ(f(y) f(x))) = f(x λ ) + f(x λ ) T (x x λ ) f(x) 0 2) Suppose f is convex and le x, d R n, hen by (b) of Par (1), f(x + d) f(x) + f(x) T d for all R Replacing he lef hand side of his inequaliy wih is second-order Taylor expansion yields he inequaliy or equivalenly, f(x) + f(x) T d dt 2 f(x)d + o( ) f(x) + f(x) T d, Leing 0 yields he inequaliy 1 2 d 2 f(x)d + o(2 ) 0 d T 2 f(x)d 0 Since d was arbirary, 2 f(x) is posiive semi-definie Conversely, if x, y R n, hen by he Mean Value Theorem here is a λ (0, 1) such ha where x λ = λy + (1 λ) x Hence f(y) = f(x) + f(x) T (y x) (y x)t 2 f(x λ )(y x) f(y) f(x) + f(x) T (y x) since 2 f(x λ ) is posiive semi-definie Therefore, f is convex by (b) of Par (1) Convexiy is also preserved by cerain operaions on convex funcions A few of hese are given below Theorem 115 Le f : R n R, h : R s R k R and f ν : R n R be convex funcions for ν N where N is an arbirary index se, and le ν i N and λ i 0, i = 1,, m Then he following funcions are also convex (1) φ f, where φ : R R is any non-decreasing funcion on R (2) f(x) := m i=1 λ if nui (x) (Non-negaive linear combinaions) (3) f(x) := max ν N f ν (x) (poinwise max) (4) f(x) := sup { m i=1 f ν i (x i ) } m x = i=1 xi (infimal convoluion) (5) f (y) := sup x R n[y T x f(x)] (convex conjugaion) (6) ψ(y) = inf x R s h(x, y) (infimal projecion)

9 70 6 OPTIMALITY CONDITIONS FOR UNCONSTRAINED PROBLEMS 141 More on he Direcional Derivaive I is a powerful fac ha convex funcion are direcionally differeniable a every poin of heir domain in every direcion Bu his is jus he beginning of he sory The direcional derivaive of a convex funcion possess several oher imporan and surprising properies We now develop a few of hese Definiion 15 Le h : R n R {+ } We say ha h is posiively homogeneous if h(λx) = λh(x) for all x R and λ > 0 We say ha h is subaddiive if h(x + y) h(x) + h(y) for all x, y R Finally, we say ha h is sublinear if i is boh posiively homogeneous and subaddiive There are numerous imporan examples of sublinear funcions (as we shall soon see), bu perhaps he mos familiar of hese is he norm x Posiive homogeneiy is obvious and subaddiiviy is simply he riangle inequaliy In a cerain sense he class of sublinear funcion is a generalizaion of norms I is also imporan o noe ha sublinear funcions are always convex funcions Indeed, given x, y dom h and 0 λ 1, h(λx + (1 λ)y) h(λx) + h(1 λ)y) = λh(x) + (1 λ)h(y) Theorem 116 Le f : R n R {+ } be a convex funcion Then a every poin x dom f he direcional derivaive f (x; d) is a sublinear funcion of he d argumen, ha is, he funcion f (x; ) : R n R {+ } is sublinear Thus, in paricular, he funcion f (x; ) is a convex funcion Remark 13 Since f is convex and x dom f, f (x; d) exiss for all d R n Proof Le x dom f, d R n, and λ > 0 Then showing ha f (x; ) is posiively homogeneous Nex le d 1, d 2 R n, Then f (x; λd) = f(x + λd) f(x) lim 0 = f(x + λd) f(x) lim λ 0 λ = λ lim (λ) 0 = λf (x; d), f(x + (λ)d) f(x) (λ) f (x; d 1 + d 2 ) = lim 0 f(x + (d 1 + d 2 )) f(x) f( 1 2 = lim (x + 2d 1) (x + 2d 2)) f(x) 0 lim 0 lim 0 showing ha f (x; ) is subaddiive and compleing he proof 1 2 f(x + 2d 1) f(x + 2d 2) f(x) 1 2 (f(x + 2d 1) f(x)) (f(x + 2d 2) f(x)) f(x + 2d 1 ) f(x) f(x + 2d 2 ) f(x) = lim + lim = f (x; d 1 ) + f (x; d 2 ),