Department of Chemical and Biological Engineering LECTURE NOTE II. Chapter 3. Function of Several Variables

Transcription

1 LECURE NOE II Chapter 3 Functon of Several Varables Unconstraned multvarable mnmzaton problem: mn f ( x), x R x N where x s a vector of desgn varables of dmenson N, and f s a scalar obectve functon - Gradent of f: f f f f f = x1 x x3 x N - Possble locatons of local optma ponts where the gradent of f s zero boundary ponts only f the feasble regon s defned ponts where f s dscontnuous ponts where the gradent of f s dscontnuous or does not exst - ssumpton for the development of optmalty crtera f and ts dervatves exst and are contnuous everywhere 31 Optmalty Crtera - Optmalty crtera are necessary to recognze the soluton - Optmalty crtera provde motvaton for most of useful methods - aylor seres expanson of f 1 f ( x) = f( x) + f( x) x+ x f( x) x+ O3 ( x) f where x s the current expanson pont, f( x) = x = x x s the change n x, x x f ( x) s the NxN symmetrc Hessan matrx at x, O ( x) 3 s the error of nd-order expanson - In order for x to be local mnmum f = f( x) f( x) for x x δ ( δ > ) - In order for x to be strct local mnmum f = f( x) f( x) > for x x δ ( δ > ) II-1

2 - Optmalty crteron (strct) 1 = + > < f f( x) f( x) f( x) x x f( x) x, x δ f x f x ( ) = and ( ) > (postve defnte) - For Qz ( ) = z z s postve defnte f Qz ( ) >, z s postve semdefnte f Qz ( ), zand z z z= s negatve defnte f Qz ( ) <, z s negatve semdefnte f Qz ( ), zand z z z= s ndefnte f Qz ( ) > for some zand Qz ( ) < for other z est for postve defnte matrces 1 If any one of dagonal elements s not postve, then s not pd ll the leadng prncpal determnants must be postve 3 ll egenvalues of are postve est for negatve defnte matrces 1 If any one of dagonal elements s not negatve, then s not nd ll the leadng prncpal determnant must have alternate sgn startng from D 1 < (D >, D 3 <, D 4 >, ) 3 ll egenvalues of are negatve est for postve semdefnte matrces 1 If any one of dagonal elements s nonnegatve, then s not psd ll the prncpal determnants are nonnegatve est for negatve semdefnte matrces 1 If any one of dagonal elements s nonpostve, then s not nsd ll the -th order prncpal determnants are nonpostve f s odd, and nonnegatve f s even Remar 1: he prncpal mnor of order of NxN matrx Q s a submatrx of sze x obtaned by deletng any n- rows and ther correspondng columns from the matrx Q Remar : he leadng prncpal mnor of order of NxN matrx Q s a submatrx of sze x obtaned by deletng the last n- rows and ther correspondng columns Remar 3: he determnant of a prncpal mnor s called the prncpal determnant For NxN matrx, there are N 1 prncpal determnant n all II-

3 - he statonary pont x s a mnmum f f ( x) s postve defnte, maxmum f f ( x) s negatve defnte, saddle pont f f ( x) s ndefnte - heorem 31 Necessary condton for a local mnmum * For x to be local mnmum of f(x), t s necessary that * * f( x ) = and f( x ) - heorem 3 Suffcent condton for strct local mnmum * * If f( x ) = and f( x ) >, * then x to be strct or solated local mnmum of f(x) Remar 1: he reverse of heorem 31 s not true (eg, f(x)=x 3 at x=) Remar : he reverse of heorem 3 s not true (eg, f(x)=x 4 at x=) 3 Drect Search Methods - Drect search methods use only functon values - For the cases where f s not avalable or may not exst Modfed smplex search method (Nelder and Mead) - In n dmensons, a regular smplex s a polyhedron composed of n+1 equdstant ponts whch form ts vertces (for -d equlateral trangle, for 3-d tetrahedron) - Let x = ( x 1, x,, xn) ( = 1,,, n+ 1) be the -th vector pont n R n of the smples vertces on each step of the search Defne f( x ) = max{ f( x ); = 1,, n+ 1}, h f( x ) = max{ f( x ); = 1,, n+ 1} and g h f( xl ) = mn{ f( x ); = 1,, n+ 1} Select an ntal smplex wth termnaton crtera (M=) ) Decde x h, x g, xl among (n+1) ponts n smplex vertces and let x c be the x 1 1 centrod of all vertces excludng the worst pont h 1 n + xc = x xh n = ) Calculate f ( x h ), f ( x l ), and f ( x g ) If xl s same as prevous one, then let M=M+1 If M>165n+5n, then M= and go to v) ) Reflecton: xr = xc + α( xc xh) (usually α =1) If f ( xl ) f( xr) f( xg), then set xh = xr and go to ) v) Expanson: If f ( x ) < f( x ), x = x + γ ( x x ) r l e c r c II-3

4 (8 γ 3 ) If f ( xe) f( xr), then set xh = xe and go to ) v) Contracton: If f ( xr) f( xh), xt = xc + β ( xh xc) (4 β 6 ) Else f f ( xr) > f( xg), xt = xc β ( xh xc) hen set xh = xt and go to ) v) If the smplex s small enough, then stop Otherwse, Reducton: x = x + 5( x x ) for = 1,,, n+ 1 nd go to ) l l Remar 1: he ndces h and l have the one of value of Remar : he termnaton crtera can be that the longest segment between ponts s small enough and the largest dfference between functon values s small enough Remar 3: If the contour of the obectve functon s severely dstorted and elongated, the search can very neffcent and fal to converge Hooe-Jeeves Pattern Search - It conssts of exploratory moves and pattern moves Select an ntal guess x, ncrement vectors for = 1,,, n and termnaton crtera Start wth =1 ) Exploratory search: ( ) ( 1) Let =1 and xb = x B ry xn = xb + If f ( xn ) < f( xb ), then xb = xn C Else, try xn = xb If f ( xn ) < f( xb ), then xb = xn D Else, let = +1 and go to B untl >n ( ) ( 1) ) If exploratory search fals ( xb = x ) * ( 1) If < ε for = 1,,, n, then x = x and stop B Else, = 5 for = 1,,, n and go to ) ) Pattern search: 1) ( 1) Let xp = xb + ( xb xb ) 1) ( ) B If f ( xp ) < f( xb ), then x = x p and go to ) C Else, x = x b and go to ) Remar 1: HJ method may be termnated prematurely n the presence of severe nonlnearty and wll degenerate to a sequence of exploratory moves Remar : For the effcency, the pattern search can be modfed to perform a lne search n the pattern search drecton Remar 3: he Rosenbloc s rotatng drecton method wll rotate the exploratory search drecton based on the prevous moves usng Gram-Schmdt orthogonalzaton II-4

5 - Let ξ1, ξ,, ξn be the ntal search drecton - Let α be the net dstance moved n ξ drecton nd u1 = α1ξ1+ αξ + + αnξn u = αξ + + αξ n n hen u = αξ n n n ˆ ξ = u / u ξ = w / w for =,3,, n where w = u [( u) ξ] ξ = 1 - Use ξ1, ξ,, ξn as a new search drecton for exploratory search Remar 4: More complcated methods can be derved However, the next Powell s Conugate Drecton Method s better f a more sophstcated algorthm s to be used Powell s Conugate Drecton Method - Motvatons It s based on the model of a quadratc obectve functon If the obectve functon of n varables s quadratc and n the form of perfect square, then the optmum can be found after exactly n sngle varable searches Quadratc functons: ( ) qx = a+ bx+ 5xC x Smlarty transform (Dagonalzaton): Fnd wth x=z so that Qx ( ) = xcx= z Cz= zd z (D s a dagonal matrx) cf) If C s dagonalzable, s the egenvector of C For optmzaton, C of obectve functon s not generally avalable - Conugate drectons Defnton: Gven an nxn symmetrc matrx C, the drecton s 1, s,,, s r (r n) are sad to be C conugate f the drectons are lnearly ndependent and s C s = for all II-5

6 Remar 1: If ss = for all they are orthogonal Remar : If s s the -th column of a matrx, then C s a dagonal matrx - Parallel subspace property For a D-quadratc functon, pc a drecton d and two ntal ponts x 1 and x Let z be the mnmum pont of mn f ( x + λd) f f x = = ( b + x C ) d = λ x λ x= z1 or z ( b + z C) d = ( b + z C) d = ( z z ) C d = 1 1 ( z z ) and d are conugate drectons 1 λ - Extended Parallel subspace property For a quadratc functon, pc n-drecton s =e ( = 1,,, n) and a ntal ponts x ) Perform a lne search n s n drecton and let the result be x 1 ) Perform n lne searches for s 1, s,, s n startng from a last lne search result Let the last pont be z 1 after n lne search ) hen replace s wth s +1 (=1,, n-1) and set s n = (z 1 -x 1 ) v) Repeat ) and ) (n-1) tmes hen s 1, s,, s n are conugate each other - Gven C, fnd n-conugate drectons Choose n lnearly ndependent vectors, u 1, u,, u n Let z 1 = u 1 1 u z z = u z for,3,, n = = 1 z z B Recursve method (from an arbtrary drecton z 1 ) z z z z z 1 1 = 1 1 z1 z1 II-6

7 z z z z z+ 1 = z z z 1 for,3,, n-1 = z z z 1z 1 cf) Select b so that z z+ 1 = z ( z + bz) = - Powell s conugate drecton method Select ntal guess x and a set of n lnearly ndependent drectons (s = e ) (1) (1) (1) ) Perform a lne search n e n drecton and let the result be x and x = x (=1) ) Startng at x, perform n lne search n s drecton from the prevous pont of lne search result for = 1,,, n Let the pont obtaned from the each lne search be x ) Form a new conugated drecton, s n+1 usng the extended parallel subspace property sn+ 1 = ( xn x )/ xn x * v) If sn+ 1 < ε, then x = x and stop v) Perform addtonal lne search n s n+1 drecton and let the result be x n + 1 v) Delete s 1 and replace s wth s +1 for = 1,,, n hen set ( + 1) ( x = x ) n + 1 and =+1 and go to ) Remar 1: If the obectve functon s quadratc, the optmum wll be found after n lne searches Remar : Before step v), needs a procedure to chec the lnear ndependence of the conugate drecton set Modfcaton by Sargent * Suppose λ s obtaned by mn f ( x λ sn + 1) λ 1) ( ) + ( x = x + λsn+ 1) nd let Chec f f( x ) f( x ) = max f( x ) f( x ) m 1 m 1 ( ) 1) * f( x ) f( x ) λ < f( xm 1) f( xm ) 5 If yes, use old drectons agan Else delete s m and add s n+1 B Modfcaton by Zangwll D s1 s s n = and Let [ ] x x = max x x m 1 m 1 Chec f x x m 1 m s n+ 1 det( D ) ε If yes, use old drectons agan Else delete s m and add s n+1 Remar 3: hs method wll converge to a local mnmum at superlnear convergence rate II-7

8 cf) Let ( + 1) ε lm = C where ε = x x r ε * If C<1, then t s convergent at r-order of convergence rate r=1 : lnear convergence rate r= : quadratc convergence rate r=1 and C= : superlnear convergence rate mong unconstraned multdmensonal drect search methods, the Powell s conugate drecton method s the most recommended method 33 Gradent based Methods - ll technques employ a smlar teraton procedure: ( + 1) ( ) ( ) ( ) x = x + α s( x ) where α s the step-length parameter found by a lne search, and sx ( ) s the search drecton - he α s decded by a lne search n the search drecton sx ( ) ) Start from an ntal guess x (=) ) Decde the search drecton sx ( ) ) Perform a lne search n the search drecton and get an mproved pont v) Chec the termnaton crtera If satsfed, then stop v) Else set =+1 and go to ) ( 1) x + - Gradent based methods requre accurate values of frst dervatve of f(x) - Second-order methods use values of second dervatve of f(x) addtonally Steepest descent Method (Cauchy s Method) f( x) = f( x) + f( x) x+ (hgher-order terms gnored) f ( x) f( x) = f( x) x he steepest descent drecton: Maxmze the decent by choosng ( ) = argmax ( ) = ( ) ( α > ) * x f x x α f x x x he search drecton: ermnaton crtera: sx ( ) ( ) ( ) = α f( x ) < and/or ( ) f( x ) ε f 1) x x / x < ε x Remar 1: hs method shows slow mprovement near optmum ( f( x) ) Remar : hs method possesses a descent property II-8

9 f x < ( ) ( ) s( x ) Newton s Method (Modfed Newton s Method) f( x) = f( x) + f( x) x+ (hgher-order terms gnored) he optmalty condton for approxmate dervatve at x : f( x) = f( x) + f( x) x= 1 x = f( x) f( x) ( ) ( ) 1 ( ) he search drecton: s( x ) = f( x ) f( x ) (Newton s method) ( ) ( ) ( ) 1 ( ) ( s x = α f x ) f( x ) (Modfed Newton s method) Remar 1: In the modfed Newton s method, the step-sze parameter α s decded by a lne search to ensure for the best mprovement Remar : he calculaton of the nverse of Hessan matrx f ( x ) mposes qute heavy computaton when the dmenson of the optmzaton varable s hgh Remar 3: he famly of Newton s methods exhbts quadratc convergence ε 1) ( ) C ε (C s related to the condton of Hessan ( ) f ( x ) ) * 1) * ( ) * f ( x ) f ( x ) x x = x x f ( x ) f x f x x x f x = f ( x ) f ( x ) * ( ) * ( ) ( ) ( ) ( x x = x x ) f ( x ) * ( ) * ( ) + ( )( ) ( ) 1 lso, f the ntal condton s chosen such that ε <, the method wll C converge It mples that the ntal condton s chosen poorly, t may dverge Remar 4: he famly of Newton s methods does not possess the descent property ( ) ( ) 1 ( ) f( x ) s( x ) = f( x ) f( x ) f( x ) < only f the Hessan s postve defnte Marquardt s Method (Marquardt s compromse) - hs method combnes steepest descent and Newton s methods - he steepest descent method has good reducton n f when x s far from * - Newton s method possesses quadratc convergence near x * x - he search drecton: sx ( ) ( ) ( ) 1 ( ) [ = H + λ I ] f( x ) - Start wth large λ, say 1 4 (steepest descent drecton) and decrease to zero II-9

10 1) ( ) 1) ( ) If f ( x ) < f( x ), then set λ = 5λ 1) ( ) Else set λ = λ Remar 1: hs s qute useful for the problems wth obectve functon form of f ( x) = f ( x) + f ( x) + + f ( x) (Levenberg-Marquardt method) 1 m Remar : Goldsten and Prce lgorthm Let δ ( δ < 5) and γ be postve numbers ) Start from x wth =1 Let φ ( x ) = f( x ) ) Chec f ε ( ) f( x ) < If yes, then stop ) Calculate ( ) f( x ) f( x θφ ( x )) (, θ ) = ( ) ( ) θ f( x ) φ( x ) gx If gx (,1) < δ, select θ such that δ < g( x, θ ) < 1 δ Else, θ = 1 v) Let Q = [ Q1 Q Q n ] (approxmaton of the Hessan) where Q f x + f x e f x = ( 1) γ f( x ) ( ) ( 1) ( ) ( γ ( ) ) ( ) If then ( x ) Q s sngular or φ ( x ) = f( x ) Else ( ) ( ) 1 ( ) f( x ) Q( x ) f( x ), φ 1 ( ) ( x ) Qx ( ) f( x ) = v) Set x = x θφ( x ) and =+1 hen go to ) 1) Conugate Gradent Method - Quadratcally convergent method: he optmum of a n-d quadratc functon can be found n approxmately n steps usng exact arthmetc - hs method generates conugate drectons usng gradent nformaton - For a quadratc functon, consder two dstnct ponts, x and x (1) Let gx ( ) = f( x ) = C x + b and (1) (1) (1) gx ( ) = f( x ) = x + b C = = = (1) (1) gx ( ) gx ( ) gx ( ) C( x x ) C x (Property of quadratc functon: expresson for a change n gradent) ( 1) ( ) ( ) ( ) - Iteratve update equaton: + x = x + α s( x ) II-1

11 f x α ( + 1) ( ) ( ) = bs + s C( x + α s ) ( ) = s ( b+ Cx ) + s α s = α = s ( ) ( ) s f( x ) ( ) ( ) Cs and ( 1) ( ) ( + f x ) s = (optmalty of lne search) - Search drecton: - In order that the 1 ( ) ( ) s = g + γ s wth = s = g s s C-conugate to all prevous search drecton ) Choose γ such that (1) s C s = where s = g + γ s = g γ g (1) (1) (1) + = (1) [ g γ g ] C[ x/ α ] x =α s ( ) (1) [ g + γ g ] g = (property of quadratc functon) (1) (1) (1) (1) (1) (1) g ( g g ) g g g γ = = = = (1) g ( g g ) g g g g g ) Choose γ and γ (1) such that s () C s (1) = and s () C s = where () () (1) (1) s = g γ g γ ( g + γ g ) γ = and (1) γ = g g () (1) ) In general, s = g + γ s ( ) (1) ( ) ( 1) ( ) g ( 1) s = g + s (Fletcher and Reeves Method) ( 1) g Remar 1: Varatons of conugate gradent method ) Mele and Cantrell (Memory gradent method) ( ) ( 1) s = f( x ) + γ s (1) ( ) ( 1) where γ s sought drectly at each teraton such that s C s = cf) Use when the obectve and gradent evaluatons are very nexpensve II-11

12 ) Danel ( 1) s f( x ) f( x ) ( 1) = ( ) + ( 1) ( ) ( 1) s f x s s f( x ) s ) Sorenson and Wolfe x ( ) gx ( ) s f x + s gx ( ) s ( ) ( ) ( 1) = ( ) ( ) ( 1) v) Pola and Rbere s f x x ( ) gx ( ) s gx ( ) ( ) ( ) ( 1) = ( ) + ( 1) Remar : hese methods are doomed to a lnear rate of convergence n the absence of perodc restarts to avod the dependency of the drectons Set s = g( x ) whenever ( ) ( 1) ( ) ( ) ( ) ( ) gx gx gx or every n teratons Remar 3: he Pola and Rbere method s more effcent for general functons and less senstve to nexact lne search than the Fletcher and Reeves Quas-Newton Method - Mmc the Newton s method usng only frst-order nformaton - Form of search drecton: sx ( ) ( ) ( ) = f( x ) where s an nxn matrx call the metrc - Varable metrc methods employ search drecton of ths form - Quas-Newton method s a varable metrc method wth the quadratc property x = C 1 - Recursve form for estmaton of the nverse of Hessan ( ) = + ( s a correcton to the current metrc) 1) c c 1 * 1 - If approaches to H = f ( x ), on addtonal lne search wll produce the mnmum f the functon s quadratc 1 ( ) - ssume H = β ( ) 1) ( ) hen x = β β ( ) c g = x / β Famly of solutons ( ) 1 x y z c = ( ) (y and z are arbtrary vectors) ( ) β y z - DFP method (Davdon-Fletcher-Powell) II-1

13 Let β = 1, y = x and g z = x ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) ( ) ( 1) x x = + ( 1) ( 1) ( 1) ( 1) ( 1) If s any symmetrc postve defnte, then wll be so n the absence of round-off error ( = I s a convenent choce) ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) z x x z z z = + ( 1) ( 1) ( 1) ( 1) ( 1) x g ( ab) ( z x ) = aa + a= z b= bb x ( ) ( 1) z z z z ( 1) ( 1) 1/ ( 1) 1/ ( 1) where, ( 1) ( 1) ) ( 1) ( 1) ( 1) ( ) ( 1) ( 1) ( 1) ( 1) x g = x g x g = x g ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) x g = ( α g g ) > ) ( aa)( bb) ( ab) (Schwarz nequalty) ) If a and b are proportonal (z and ( 1) g are too), ( aa )( bb ) ( ab ) = but ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) x z = c x g = cα g g z z > hs method has the descent property ( ) f = f( x ) x= α f( x ) f( x ) < for ( α ) > - Varatons McCormc (Pearson No) ( 1) ( 1) ( 1) ( 1) ( ) ( 1) ( x ) x = + ( 1) ( 1) x Pearson (Pearson No3) ( ) ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) ( x ) = + Broydon 1965 method (not symmetrc) ( 1) ( 1) ( 1) II-13

14 ( ) ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) ( x ) x = + x ( 1) ( 1) ( 1) Broydon symmetrc Ran-one method (1967) Zoutend = + ( ) ( 1) = ( ) ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) ( x )( x ) ( x ) ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) BFS method (Broydon-Fletcher-Shanno, ran-two method) I I x x x ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) ( ) x ( 1) x x x = ( 1) ( 1) ( 1) + ( 1) ( 1) ( 1) Invarant DFP (Oren, 1974) ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) ( ) x ( 1) x x = ( 1) ( 1) ( 1) ( 1) + ( 1) ( 1) ( 1) ( 1) x Hwang (Unfcaton of many varatons) ( ) ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) ( 1) = + x B x where B s x and Remar: If ω = 1 and ( 1) ( 1) ( 1) ( 1) ( 1) B x g = [ ω -1] B = dag(1/ x, 1/ ), ths method wll be same as DFP method Remar 1: s these methods terate, tends to become ll-condtoned or nearly sngular hus, they requre restart ( = I: loss of nd -order nformaton) cf) Condton number= rato of max and mn magntudes of egenvalues of Ill-condtoned: f has large condton number Remar : he sze of s qute bg f n s large (computaton and storage) Remar 3: BFS method s wdely used and nown that t has decreased need for restart and t s less dependent on exact lne search Remar 4: he lne search s the most tme-consumng phase of these methods Remar 5: If the gradent s not explctly avalable, the numercal gradent can be obtaned usng, for example, forward and central dfference approxmatons If II-14

15 the changes n x and/or f between teratons are small, the central dfference approxmaton s better at the cost of more computaton 34 Comparson of Methods - est functons Rosenbloc s functon: f ( x) = 1( x x ) + (1 x ) x x1x + 1 Fenton and Eason s Functon: f( x) = 1+ x x 1 ( xx 1 ) Wood s functon: f ( x) = 1( x x ) + (1 x ) + 9( x x ) + (1 x ) ( x 1) + ( x4 1) + 198( x 1)( x4 1) - est results Hmmelblau (197): BFS, DFP and Powell s drect search methods are superor Sargent and Sebastan (1971): BFS among BFS, DFP and FR methods Shanno and Phua (198): BFS Relats (1983): FR among Cauchy, FR, DFP, and BFS methods II-15