ALGORITHMS FOR OPTIMAL DECISIONS 2. PRIMAL-DUAL INTERIOR POINT LP ALGORITHMS 3. QUADRATIC PROGRAMMING ALGORITHMS & PORTFOLIO OPTIMISATION

Transcription

1 - (00) Itroducto - ALGORIHMS FOR OPIMAL DECISIONS. INRODUCION: ì Nolear decso problems ì Basc cocepts ad optmalty ì Basc Algorthms: Steepest descet; Frak-Wolfe; Barrer/SUM. PRIMAL-DUAL INERIOR POIN LP ALGORIHMS 3. QUADRAIC PROGRAMMING ALGORIHMS & PORFOLIO OPIMISAION 4. SQP & INERIOR POIN ALGORIHMS FOR NLP ether 5. ALGORIHMS FOR GAMES; EQUILIBRIA & NONLINEAR SYSEMS OF EQUAIONS or 6. NONLINEAR PROGRAMMING ALGORIHMS FOR CONVEX CONSAINS Suggested referece books: Nolear Programmg - D. Bertsekas (Athea) Practcal Methods of Optmzato - R. Fletcher (Wley) Lear & Nolear Programmg - D.G. Lueberger & Y. Ye (Sprger) Numercal Optmzato - J. Nocedal & S. Wrght (Sprger)

2 - (00) Itroducto - INRODUCION. HE GENERAL NONLINEAR PROGRAMMING (NLP) PROBLEM: m f (x) h (x) Ÿ 0 ; g (x) = 0 or max f (x) h (x) Ÿ 0 ; g (x) = 0 x h(x) g (x) x h(x) g (x) x = ; h (x) = ; g (x) = ã ã ã x h(x) e g (x) Feasble rego: e = x g (x) = 0 ; h (x) Ÿ 0 Optmalty of x for m of f (x) Ê x e ad f (x ) Ÿ f (x); ax e Optmalty of x for max of f (x) Ê x e ad f (x) f (x); ax e f, g ad h lear Ê LP problem Example : o maufacture a product costs c/ut. It sells at prce p/ut. Customer demad s d(p) uts. Fd the prce to charge to maxmse profts. he proft s (p - c) d(p). he decso varable s p. Problem: max p (p - c) d(p). Example : If k uts of captal ad j uts of labour are used, a compay ca maufacture (k j) uts of a product. Captal ca be purchased at 4/ut ad labour at /ut. A total of 8 s avalable to purchase k ad j. How ca the compay maxmse the uts of the product that ca be maufactured? We wsh to solve max k j 4 k + jÿ 8; k 0; j 0. Remark : Eve f e s a covex set, the optmal soluto of a NLP s ulkely to be at a extreme pot (vertex) of e. I example, the feasble rego (bouded by the tragle ABC) ad the costat proft curves k j =, k j = ad k j = 4. he optmal soluto occurs where the costat proft curve s taget to the boudary of e. hus the

3

4 - (00) Itroducto 3 - optmal k j = 4, k =, j = 4 (pot D). Of course, pot D s ot a extreme pot of e, because the costat proft les are ot straght. FIGURE. Remark : he optmal soluto of a NLP may ot be eve o the boudary of e. $ For example, cosder max x x % 0 Ÿ x Ÿ. he optmal soluto s at x whch s clearly ot o the boudary of e. FIGURE œ " # Example 3: Furture House (FH) has 700 uts of wood (W) for maufacturg tables () ad chars (C). Small productos requre uts of W ut of W C ;. For small productos, FH's dstrbutor wll pay 0/ ad 5/C. Let x, x deote respectvely the umber of ad C produced. For larger quattes of 's ad C's more precse maufacturg methods become worthwhle. hs ecoomes of scale (cludg savg ad reusg shavgs, scraps etc) makes W used depedet o x, x. Suppose uts of W used 0.5 = x uts of W used 0.5 C x x = FH's dstrbutor requres twce as may C's as 's. For large orders, t requres a cetve dscout of 0.% for every or C, so that prce prce C (x + x ) = 0 ( 0.00) ; (x + x ) = 5 ( 0.00). How may 's ad C's should FH maufacture to maxmse ts reveue? max x, x f ( x, x ) = 0 x + 5 x subject to (x + x ) x x x h (x, x ) = x.5+ + x Ÿ 0 h(x, x) = x x Ÿ 0 3 h (x, x ) = x Ÿ 0; h (x, x ) = x Ÿ 0 hs problem has = varables ad m = 4 costrats. FIGURE 3. LOCAL MAXIMA OR MINIMA x x x x For ay NLP, x = e s a local maxmum f, ax = e ã ã x x

5

6 - (00) Itroducto 4 - ad ± x x ± ( =,..., ) % for some % 0, f (x ) f (x). hus, x s a local maxmum f f (x) f (x) holds a feasble x that are close to x. Local mmum s defed smlarly for f (x ) Ÿ f (x) holds a feasble x close to x. 3. CONVEX & CONCAVE FUNCIONS A set f s covex f x', x'' f Ê α x' + ( α) x'' f, 0 Ÿ α Ÿ. A fucto f (x) s covex o f f for x', x'' f, 0 Ÿ α Ÿ f ( α x' + ( α) x'') Ÿ α f(x') + ( α) f(x'') [the le jog x' ad x'' s ever below the fucto value at that pot]. A fucto f(x) cocave o f f for x', x'' f, 0 Ÿ α Ÿ s f ( α x' + ( α) x'') α f(x') + ( α) f(x'') Ê f(x) s covex f ad oly f f(x) s cocave. FIGURES 4 & 5 Remark: For a lear fucto, f(x) = a x + b, equalty follows so t s both covex ad cocave: a α x' + ( α) x'' + b = αa x' + αb + ( α) a x'' ( α) b. " x Example 4: For x 0, f(x) = x ad f(x) = e are covex; f(x) = x # s cocave. FIGURE 6 Example 5: It ca be show that the sum of two covex (cocave) fuctos s covex x (cocave). hus, f(x) = x e s covex. heorem: For covex f ad covex f(x), the local mmum s also the global mmum, for cocave f(x), the local maxmum s also the global maxmum. Proof Suppose that x s a local mmum. Suppose also that there s aother pot: x f, x Á x, wth f(x) f(x).

7

8

9 - (00) Itroducto 5 - he, for we have α x + ( α) x, α (0, ), f( α x + ( α) x ) Ÿ αf (x ) + ( α) f (x ) f (x ) whch cotradcts that x s a local mmum pot. Let ` ` x" ` f # ` f f f ` = ` f x Þ $ ã f ` ` f x heorem: Let f(x) be oce cotuously dfferetable. he, f(x) s covex over a covex set f f ad oly f. f(y) f(x) + ff (x)(y x) ax, y f. Proof We omt the proof (see e.g. Lueberger ad Ye p. 95) but motvate by otg that the taget to f(y), at x, s alwas below the covex fucto f(y). Defto: he Hessa of f(x) at x, s the symmetrc matrx H (x,..., x ) whose j th etry s H (x,..., x ) =. j ` f (x) j 6 x " 3 # Example 6: f(x, x ) = (x ) + x x + (x ), H (x, x ) =. Defto: H(x) s postve (egatve) sem-defte f for 0 Á v, v H(x) v ( Ÿ ) 0. Furthermore, H(x) s postve (egatve) defte f for 0 Á v, v H(x) v ( ) 0. heorem: Suppose f (x) has secod partal dervatves for all x f where f has cotas a teror pot. he, f (x) s a covex (cocave) fucto f ad oly f a x f, H (x) s postve (egatve) sem-defte. Proof

10 - (00) Itroducto 6 - By aylor expaso we have: " # α f(y) = f(x) + ff (x)(y x) + (y x) H(x + (y x))(y x) for some α, 0 Ÿ α Ÿ. Clearly, f the Hessa s postve everywhere sem-defte, we have f(y) f(x) + ff (x)(y x) whch, by the above theorem, mples that f(x) s covex. Suppose the Hessa s ot postve sem-defte at some pot x f. By cotuty of the Hessa t ca be assumed that x s a teror pot of f. here s a y f such that (y x) H(x)(y x) 0. Aga, by cotuty of the Hessa, y may be selected so that for all α, 0 Ÿ α Ÿ, (y x) H(x + α(y x))(y x) 0. hs, vew of the aylor expaso above mples that f(y) f(x) + ff (x)(y x) does ot hold ad cosequetly, vew of the prevous theorem, f s ot covex. have " # Example 7: f(x, x ) = (x ) + x x + (x ) s a covex fucto o f =. We `f(x, x) x + x f f(x, x ) œ `f(x, x) œ ; x + x H(x, x) = H s postve sem-defte (each prcpal mor s oegatve). Exercses:. Show that the tersecto \ of ay umber of covex sets f s covex (e \ œ f s covex. [follows by defto of covex f.]. Show that f f(x) ad g(x) are covex fuctos o a covex set f, the f(x) + g(x) s a covex fucto o f. 3. Show that f f s a covex (cocave) fucto, x f (x) Ÿ b s a covex (cocave) set. 4. UNCONSRAINED OPIMISAION

11 - (00) Itroducto 7 - m f (x) x or max f (x) x Defto: A pot x for whch f f (x) = 0 s a statoary pot of f(x). If H(x ) s also postve (egatve) defte, the x s a local m (max) of f(x). Remark. Covex (cocave) f (x) Ê x s a global mmum (maxmum) (see heorem Secto 3). Example 8: A moopolst produces a product for two types of customers (C & C). If q uts are produced for C, the C s wllg to pay (70 4q )/ut. For C, ths s (50 5q )/ut. Maufacturg cost for uts s (q + q ) (q +q ). o max proft, how much should be sold to each customer? Objectve proft fucto: f (q, q ) = q (70 4q ) + q (50 5q ) 00 5 (q + q ) `f(q, q) 70 8 q 5 Ê q = ` q 8 f f(q, q ) œ `f(q, q) œ = 0; q 5 9 Ê q = ` q thus, we have foud the statoary pot of f. Next we exame the Hessa 55 H(q, q) = whch s egatve defte. hus, q =, q = maxmses f ad yelds the proft f(, ) = ( ) ) œ 39.8 Exercse: A compay maufactures two products. If t charges a prce p for product, t ca sell q uts of product where q = 60 3 p + p ad q = 80 p + p. It costs 5/ut to produce product ad 7/ut to produce product. I order to maxmse proft, how much of each product should be produced? 5. OPIMALIY: LAGRANGE MULIPLIERS For the equalty costraed optmsato problem

12 - (00) Itroducto 8 - m f (x) g (x) = 0 or max f (x) g (x) = 0 g(x)=0 g(x)=0 3 g(x)=0 ã e g(x)=0 he Lagraga s defed as L (x, -) œ f (x) + -, g(x) f (x) + - g(x) f (x) + - g (x) e he costraed mmum (maxmum) s the statoary pot (x, -) of L wth respect to (x, -) f x, e ` f (x) g (x ) x ` j ` - j = - L (x, - ) = = g (x ) ad t s ecessary that the Hessa of the Lagraga = f x - L(x, ) = ` L(x, -) `x`xj s postve (egatve) sem-defte the tersecto of the costrats: g(x ) j v f L(x, - ) v ( Ÿ ) 0; av: ` v = 0; a =,...,. x j= j e Furthermore, t s suffcet that f x L(x, -) s postve (egatve) defte for all v satsfyg: ` g(x ) j= j j v = 0; a =,..., e. For covex (cocave) f ad for g(x) lear, the statoary pot of L s esured to be a mmum (maxmum) as the codto o the Hessa s satsfed. he varable vector - (lke dual varables LP) are shadow prces. he th shadow prce (e elemet of -) correspods to the th costrat. If the rght had sde of the th costrat s creased by a small amout, say? ( ether a max or m problem), the the optmal objectve fucto value wll crease by approxmately?-. Example 9: A compay s plag to sped 0, 000 o advertsg. It costs 3000/mute to advertse o V ad 000/mute to advertse o rado. If the compay buys x ms of V ad y ms of rado advertsg, ts reveue s gve by f (x, y) = x y + xy + 8x +3y

13 - (00) Itroducto 9 - NLP: maxmse f (x, y) = x y + xy + 8x +3y subject to: 3 x + y = 0 Lagraga: L(x, y, -) = x y + xy + 8x +3y + - (0 3x y) Statoary pot: `L(x, y, -) 4 x + y L(x, y, -) ` fx, y, - L(x, y, -) œ ` y œ y + x = 0; `L(x, y, -) 0 3 x y ` - y = 4 x ; x = y ; 3 x + y = 0 y = (- 3 + y) = y Ê y = x œ œ = 0 Ê - = Ê y = = Ê x = 0 7 `L `L `x `x `y 4 H(x, y) = `L `L = `y `x `y he Hessa of f (x, y) s egatve defte; f(x, y) s cocave. he costrat s lear, so the soluto obtaed maxmses f(x, y). he frm should purchase 69/8 ms of V ad 73/8 ms of rado tme. Sce - = /4, spedg a extra? 's o advertsg (for small?) would crease the compay's reveues by approxmately 0.5?. Exercse: Labour costs /hour ad captal costs /ut. If j of labour ad K hours of /3 /3 captal are avalable, the j K maches ca be produced. If the budget for purchasg captal ad labour s 0, what s the maxmum umber of maches that ca be produced? What s the mmum cost method for producg 6 maches? 6. OPIMALIY: KARUSH-KUHN-UCKER (KK) CONDIIONS m f (x) h (x) Ÿ 0 or max f (x) h (x) Ÿ 0 h (x) Ÿ 0 (or h (x) Ÿ b Í h (x) b Ÿ!) h (x) Ÿ 0 ã

14 - (00) Itroducto 0 - h (x) Ÿ 0. ñ A costrat the form h (x) b must be wrtte as h (x) + b Ÿ 0 (eg x + x should be rewrtte as x x + Ÿ 0). ña equalty costrat ca be replaced by a Ÿ ad costrat (eg x + x œ 0 s replaced by x + x Ÿ 0 ad x + x, or x x + Ÿ 0). Alteratvely, equalty costrats are treated as secto 5. (he geeral approach s Bertsekas, Nolear Programmg; Fletcher, Practcal Optmsato; Lueberger, Nolear Programmg; Rustem, Algorthms for Nolear Programmg & Multple Objectve Decsos). he Lagraga s defed as L (x,.) œ f (x) +., h(x) f (x) +. h(x) f (x) +. h (x) For f (x) ad h (x) covex (local mmum Ê global mmum) ad for a mmsato problem, the optmal soluto satsfes the KK optmalty codtos: =. f (x ) + h (x ). = 0 ` f (x) ` h (x) j j f f =. j=,..., ; =,..., ; h ( x ) = 0; h (x ) Ÿ 0; 0.. Whe the covexty assumpto s ot ecessarly statsfed at all pots x for f ad h, the t s ecessary to formalse the secod order codto that esures the maxmum or mmum. hs s smlar to the equalty costraed case ad t s ecessary that the Hessa f x. L(x, ) = ` L(x,.) `x`xj s postve (egatve) sem-defte for all v: ` h(x ) j= j j v = 0; a h (x ) = 0. Note that the latter codto s related to the equalty costrats satsfed as equaltes at the soluto. Furthermore, t s suffcet that f x L(x,.) s postve (egatve) defte the tersecto of the actve costrats: h(x ) j v f L(x,. ) v ( ) 0; av: ` v = 0; ah (x ) = 0 x j j=

15 - (00) Itroducto - he multpler. s the shadow prce for the costrat. We show ow that. s the sestvty of the optmal value of the objectve fucto to the chage the rght had sde of th th the costrat. For smplcty, we gore the superscrpt for the costrat. As before, let f f th ` f f h ` h From the optmalty codtos, we have f f(x ) + f h (x). = 0. Let c be a arbtrary value for the rght had sde of costrat h ` h ` c h (x) = c Ê = I ad let x = x(c) (.e. x satsfyg h(x)=c s a fucto of c) all evaluated at x ad c = 0. But ` h (c) ` h (c) ` c ` c ` c I = = = f h(x ) ` f (c) ` f (c) ` c ` c ` c = = f f (x ) so f f(x ) = f h (x)., ` f (c) (c) ` c ` c ` c = f f (x ) = fh (x ). =. th If the rght had of the costrat s creased from 0 to? uts (for small?), the the optmal objectve fucto value s creased by.? (for a more rgorous dscusso of shadow prces/lagrage multplers/sestvty see Lueberger ad Ye, p. 340). he codto. h ( x ) = 0 s a geeralsato of the complmetary slackess codtos of LP's. he codto mples that f. 0, the h ( x ) = 0 f h ( x ) 0, the. = 0 th ( costrat actve). th ( costat actve) Suppose that the costrat h (x) Ÿ 0 s a costrat about the use of a resource. he the frst mplcato states that f a addtoal ut of the resource assocated wth the th costrat s to have ay value, the curret optmal soluto must use all the avalable uts of the resource, drvg the costrat to ts boud. O the other had, the secod mplcato th th states that f some of the resource s uused, the addtoal amouts of the resource th have o value. If for? 0, we crease the rght had sde of the costrat from 0 to?,

16 - (00) Itroducto - the optmal objectve value must decrease or stay the same, because the crease expads the feasble rego. Example 0: For cocave f(x), descrbe the optmal soluto to max f (x) a Ÿ x Ÿ b or, alteratvely, m f (x) a Ÿ x Ÿ b. ñif f f (x) exsts the terval [ a, b], the optmal soluto wll ether be at a pot x (a, b) or at a or b. ñif the soluto s at some x (a, b) the ff (x) = 0Þ ñif the soluto s at a the f f (a) Ÿ 0 À for f f (a) 0, movg a small dstace away from a would be charactersed by f (y) = ~ f (a) + f f (a) (y - a) ad f (y) f (a), for y a, thereby cotradctg the maxmal ature of f(a). We also ote that y a s ot feasble. ñif the soluto s at b the f f (b) 0. We wrte the problem as m f (x) x b Ÿ 0; x + a Ÿ 0 ad the optmalty codtos yeld ff (x) +.. = 0. (x b) = 0. ( x + a) = 0.,. 0 x b Ÿ 0 x + a Ÿ 0 - If. =. = 0, we have f f (x) = 0. - If. = 0,. 0, the latter yelds from. ( x + a) = 0, x = a. ff (x ) +.. = 0 yelds ff (x ) œ. Ê ff (x ) 0. - If. = 0,. 0,. (x b) = 0 Ê x = b ad ff (x ) œ., ff (x ) 0Þ - If. 0,. 0 we have x = a ad x = b whch caot occur. Example : A moopolst ca purchase up to 7.5 oz of chemcal C for 0/oz. At a cost of 3/oz C ca be processed to a oz of P ad at a cost of 5/oz C ca be processed to a oz of P. If x oz of P are produced,

17 - (00) Itroducto 3 - If x oz of P are produced, (30 x ) oz P sells at. (50 x ) oz P sells at. How ca the moopolst maxmse proft? x = oz of P produced x = oz of P produced 3 x = oz of chemcal processed max x (30 x ) + x (50 x ) 3 x 5 x 0 x x + x x Ÿ 0; 3 x 7.5 Ÿ 0 3 Also x, x, x 0, but as the optmal soluto happes to satsfy these, we choose to avod ths complexty. he objectve s cocave, the costrats are lear, hece covex. So the optmal soluto s determed by the frst order KK whch are (wrtte for the mmsato of the egatve of the objectve): 30 + x = 0; x = 0; = (x + x x ) = 0; (x 7.5) = 0;., If. =. = 0, the = 0 s volated. So ths caot occur. - If. œ 0,. 0, we ote that = 0 would mply. = 0 whch would volate. 0. hus, ths case caot occur as well. - If. 0,. = 0, = 0 mples. = x = 0 yelds x = 8.5 ad x = 0 yelds x = From. (x + x x 3 ) = 0, 3 3 we obta x + x x = 0, ad hece x = 7.5. We eed ot cosder the case.,. 0 sce we have determed the soluto. For? small,. = 0 dcates that f a extra? oz of the chemcal were obtaed at o cost, the profts would crease by 0?. As. = 0, the rght to purchase a extra? oz of chemcal would ot crease profts. Exercse: Use the KK codtos to solve m (x ) + (x ) x x = 0; x x Ÿ 0; x, x 0.

18 - (00) Itroducto 4-7. HE SEEPES DESCEN ALGORIHM Suppose we wsh to solve the ucostraed problem: m f (x) x. If f (x) s a covex fucto, the the optmal soluto wll occur at a statoary pot `f(x,..., x ) f f (x) = ff(x,..., x ) œ ã = 0 `f(x,..., x ) I prevous examples, t was easy to determe the optmum solutos. For may fuctos, t may be dffcult to fd a statoary pot satsfyg optmalty. We dscuss the steepest descet algorthm (Cauchy) as the frst teratve method for computg (approxmately) optmal solutos. v v Gve a vector v =, the legth ( ) of v s gve by ã orm v v = vv = (v ") + (v#) (v ) Ay -dmesoal vector represets a drecto. For ay drecto, there are fte 3 umber of vectors represetg that drecto (eg,, represet the drecto 3 potg to a postve 45 agle ). For ay vector v, the ormalsed vector has a v v legth of ad defes the same drecto as v. hus, wth ay drecto, we may assocate a vector a ( ut) vector of legth (eg the drecto defed by v = wth v = s assocated wth the ut vector v v v v = =. For ay vector v, the ut vector s called the ormalsed v. We shall use the ormalsed 3 verso of drectos. he drecto defed by,, wll be descrbed by 3 the ormalsed vector

19 - (00) Itroducto 5 -. Cosder the fucto whose frst partal dervatves exst. he gradet of the fucto s gve by f f (x) ad we defe the steepest descet drecto f f (x) f f (x) d =. " # Example : For f (x, x ) = (x ) + (x ) x 6 f f (3, 4).6 f f (x) =, f f (3, 4) =, ff (3, 4) = 0 ad =. x 8 f f (3, 4).8 At ay pot x ^ whch les o the curve f (x,..., x ) = f (x), ^ the vector d (x) ^ = f f f (x) ^ f (x) ^ wll be perpedcular to the curve f (x,..., x ) = f (x). ^ Example 3: " f (x, x ) = (x ) # + (x ) ; at (3, 4), f.6 d (3, 4) = f (3, 4) f f (3, 4) =.8 " # s perpedcular (orthogoal) to (x ) + (x ) = 5 FIGURE 7. If x s altered by a small amout 7, the value of f (x) wll alter by approxmately 7 ` f(x) `f(x) (by defto of ). Suppose we move from a pot x, a small legth 7 a drecto defed by ormalsed vector?. By how much does f(x) chage? he quatty d f (x + 7? ) d 7 7=0 =? f f (x). dcates a frst order approxmato for the amout of chage. If f f (x) ^ f f (x) ^? =, the clearly,? ff (x) 0, ad movg the drecto 7 0 wll decrease f. " # Example 4: f (x,x ) = (x ) + (x ), at (3, 4). Heceß 6 ff (3,4) =. 8

20

21 - (00) Itroducto 6 - If we move by a step 7 the drecto? =, f wll chage (crease) by 6 8 7? f f(x) = 7 +. o show that the steepest descet drecto reduces the objectve fucto, cosder the secod f f (x) f f (x) order aylor seres expaso of f x 7 about x: f x 7 f f (x) f f (x) - = f (x) 7 f f f (x) + f f (x) 7 f (x) H f f (x). ff (x) ff (x) ff (x) - f f(x) where H s the Hessa at some mea value betwee x ad (x 7 ): For f f (x) - f f(x) H œ Hx + α(x 7 ) xfor some α, 0 Ÿ α Ÿ. f f (x) 7 small eough, the last term s eglgble. Hece, we have the descet property 7 f f(x) f f(x) ². f x f(x) he steepest descet algorthm Step 0: Startg at ay pot x 0, set ( (0, ), # (0, ), k = 0. Step : Evaluate ff (x k) ; f f f (x k) Ÿ macheps, the optmum has bee acheved, Stop. Else, evaluate f f (x k) f f (x ). k j Step : Determe the stepsze 7k as the value largest value 7 = (#), j = 0,,,... (or equvaletly the smallest value of j) for whch the equalty ff(x ) ff(x ) k f f(x ) ² k f f(x ) ² k k k f x 7 ) f ( x ) Ÿ ( 7 ff(x ) k k s satsfed. he equalty above s kow as the Armjo stepsze rule ad esures suffcet decrease of the objectve alog the steepest descet drecto. Set x = x 7 k+ k k f f(x k) f f(x ) ² k Step 3: Set k = k +, go to Step. A earler alteratve to the Armjo rule was the soluto of the problem m f f(x k) 7 0 k 7 f f(x ) ² f x. k

22 - (00) Itroducto 7 - hs uvarate mmsato s, geeral, a fte procedure whereas the ca be acheved a fte umber of reductos of the stepsze through Armjo rule j 7 = (#), j = 0,,,.... Example 5: Use the method of steepest descet to solve Choose arbtrarly x = (, ). We have 0 m f(x, x ) = (x 3) (x ) f f(x, x ) = (x 3) f ; hece f(, ) = 4 (x ). Accordg to we must choose 7 m 0 k 7 f k 7 f x f(x ), to mmse the oe dmesoal fucto f(x 7 f f(x )): k 4 f + = f ( + 4, + ) = (4 ) + ( ) k whch s mmsed where 4 f7 f + 7 = 8 (47 ) + 4 ( 7 ) = 0. Hece, 7 = 0.5. he ew pot s 4 3 x = +.5 =. As f f (3, ) = 0 ad f(x) s covex (Hessa!), we have arrved at a optmum. Example 5 s deceptve. he steepest descet algorthm coverges to a optmum oe step extremely rarely. Its covergece s very slow. It s used oly very large problems where other methods mght be approprate. We meto ths algorthm because t s the smplest, hece useful to motvate more complex algorthms. Exercse:. For ay x ( Á 0), show that x/ ² x ² has ut legth.. Use steepest descet to solve the problem m f(x, x ) = (x ) + x + (x # ). Beg at (.5,.5)

23 - (00) Itroducto 8 - " # 3. Wrte a program to solve m f(x, x) = xx x + (x ) + (x ) ad observe how slow progress may become. Beg at (.5,.5). 4. Show that the steepest descet drecto f (x ) s the soluto of f f f (x ) m d k f f (x ) d d =. Hece, the steepest descet drecto s based o the lear approxmato to the orgal problem sce the soluto of the above s the same as m d k k f (x ) + f f (x ) d d =. where f (x k) s costat. 8. CONVERGENCE A sequece of vectors x 0, x,..., x k,..., f, deoted by {x k}, s sad to coverge to the lmt x f f x k x Ä 0 as k Ä (that s, gve % 0, there s a N such that k N mples x k x %). If {x k} coverges to x, we wrte {x k} Ä x. he ope ball aroud x s a set of the form { y y x %}, for some % 0. A subset f s ope f aroud every pot x f, there s a ope ball cotaed f (e for x f, there s a % 0 such that y x %, wth y f). For example, the ball { y y %} s ope. he teror of ay set f s the set of pots x f whch are at the cetre of some ball cotaed f. A set s sad to be ope f all ts pots are all teror pots. hus, the teror of a set s always ope. he teror of the ball s the ope ball k { y y Ÿ %} { y y < %}. A pot x s a lmt of {x k} f there s a subsequece of {x k} coverget to x. he subsequece s deoted terms of the subset ^ of the postve tegers such that {x k} Ä x for k ^. A related cocept s a accumulato pot: f f ad x, the x s called a accumulato pot of f f every ball aroud x cotas at least oe pot of f dstct from x. Example 6: he set of umbers /, =,, 3,..., has zero as a accumulato pot. A set f s closed f ts complemet e f s ope. Alteratvely, a set f s closed f every pot arbtrarly close to f, s a member of f (.e. f s closed f {x k} f ad {x k} Ä x mply that x f ). For example, the ball { y y Ÿ %} s closed. he closure of ay set f s the smallest closed set cotag f. he boudary of a set s that part of the set that s ot ts teror. A set s bouded f t les etrely wth a ball of some radus % 0, { y y x Ÿ % }. A set s compact f ad oly f t s closed ad bouded.

24 - (00) Itroducto 9 - For a compact set f, the Bolzao-Weerstrass theorem (Apostol, 98) ca be used to show that every fte subset of f has a accumulato pot f. hus, f {x k } f (e each member of the sequece s f, ) the {x k} has a lmt pot f. hs establshes that there s a subsequece of {x k} covergg to a pot f. Examples of ope, closed, bouded: ope: x + x ; closed: x + x Ÿ ; & bouded x + x Ÿ, x 0, x 0 he covergece theory assocated wth the algorthms we are cocered wth depeds o the above result. A more geeral result s gve Lueberger (984; Secto 6.6) for algorthms vewed as pot-to-set mappgs. We are cocered wth algorthms that am to solve for a fxed optmal pot whle geeratg a sequece {x k} wth a compact set f such that a certa descet property s satsfed for a mert fucto (e.g. f (x k+) f (x k) ad for x solvg the problem, f (x ) Ÿ f (x ), ak). k 9. EXENSION O LINEARLY CONSRAINED PROBLEMS he smple exteso of the steepest descet drecto to lear costrats s kow as the Frak-Wolfe algorthm. Cosder m f(x) A x Ÿ b ; x 0. ad a frst order aylor seres expaso of f (x) aroud x k We have (where x s ay value for x). k k ` f(x ) j k k k k j= k j j f (x) f (x ) + (x x ) = f (x ) + ff (x ) (x x ). Sce f(x k) ad f f(x k) x k have fxed values, they do ot affect the posto of the optmum ad ca be dropped to gve a lear objectve: f f(x ) x. Let x = arg m f(x ) + ff(x ) (x x ) A x Ÿ b ; x 0 k LP k k k k k k gorg the costat terms f(x ) ad ff(x ) x, we have x LP = arg m ff(x k) x A x Ÿ b ; x 0. he lear objectve fucto decreases steadly as oe moves alog the le segmet from x k to x LP (whch s o the boudary of the feasble rego). However, the lear approxmato to the olear objectve may ot be partcularly accurate for x far away from x k. So, the olear objectve may ot cotue to decrease all the way from x k to x LP. herefore, rather tha just acceptg x LP the ext tral soluto, we ca choose the pot that mmses the

25 - (00) Itroducto 0 - olear objectve fucto alog ths le segmet. hs pot ca be chose usg uvarate mmsato (as the steepest descet algorthm). he oe varable for ths purpose s the stepsze 7 to be take from x k to x LP. I the statemet of the algorthm below, however, we have used a stepsze rule that volves a fte umber of operatos. he pot thus obtaed becomes the ew tral soluto for tatg the ext terato of the algorthm. he sequece of tral solutos geerated by repeated teratos coverges to a optmal soluto of the orgal problem. he algorthm stops as soo as the successve tral solutos are close eough together to have essetally reached ths optmal soluto. he Frak-Wolfe algorthm Step 0: Startg at a feasble tal pot x 0, choose # (0, ), set k = 0. Step : Evaluate f f (x k). Step : Determe the drecto of search/descet x x by solvg x LP = arg m ff(x k) x A x Ÿ b ; x 0. LP k x x LP k If x Ÿ macheps, stop. We have reached the optmum. k j Step 3: Determe the stepsze 7k as the value largest value 7 = (#), j = 0,,,... (or equvaletly the smallest value of j) for whch the equalty k LP k k k LP k f x + 7 (x x ) f ( x ) Ÿ ( 7 ff(x ) (x x ) " where ( (0, #). he equalty above s the Armjo stepsze rule ad esures suffcet decrease of the objectve alog the steepest descet drecto. Set x k+ = x k + 7k (x LP x k) Step 4: Set k = k +, go to Step. he algorthm coverges to the soluto as the steepest descet method. Covergece s slow, however. For steepest descet, the estmated rate s R-lear covergece k/ x x Ÿ C " ; C ad 0 ". k ad for the Frak-Wolfe algorthm x x Ÿ k C " ; C ad 0 ". k he above s the geerc defto of R-lear covergece. We shall study the covergece detals of oe algorthm as a model for such results. Example 7: " m f(x) = (x ) # + (x ) 5 x 8 x 3 x + x Ÿ 6; x, x 0 Note that the ucostraed mmum s defed by the codtos

26 - (00) Itroducto - ` f (x) ` f (x) = x 5 = 0; = 4 x 8 = 0 whch are satsfed at 5 x = u. Clearly, x volates the frst costrat. hus, more work s eeded to fd the costraed u mmum. 0 Because x = s clearly feasble (ad correspods to the tal BFS for the LP 0 costrats), let us choose t as x 0 to start the algorthm. For ths x 0, step, we have So, the LP problem Step s ` f (x) ` f (x) = 5 ; = 8. m 5 x 8 x 3 x + x Ÿ 6; x, x 0. 0 We ca use the smplex algorthm or a graphcal soluto to see that x LP =. For Step 3, 3 we eed to determe the stepsze: we depart from the algorthm descrpto to mplemet a older dea - uvarate mmsato o the le segmet x LP x 0 (.e. 0 Ÿ 7 Ÿ ). hus, we have x = x (x LP x) 0 x( 7) = + = x( ) substtutg ths the objectve fucto yelds a fucto of the sgle varable 7 x( 7) f( ) = f (0, 3 ) = 8 (3 ) + (3 ) = x( 7) hs fucto s covex ad the value of 7 (0 Ÿ 7 Ÿ ) that mmses t ths case s gve by the soluto of so that 7 0 = 3. Hece, we have x( 7) ` f( ) x( 7) ` 7 = = 0 x (/3) x = x (/3) = = + = x (/3) he ext terato volves the evaluatos ` f (x) ` f (x) = 5 ; = () = 0

27

28 - (00) Itroducto - ad x LP = arg m 5 x 3 x + x Ÿ 6; x, x 0. wth uvarate mmsato of x LP = 0 x ( 7) = x + 7 (x x ) ; where 0 Ÿ 7 Ÿ LP x( 7) = + = x ( ) f ( ) = 5 ( ) + ( ) 8 ( ) ( ) = Hece 7 = ad x( 7) ` f( ) x( 7) ` 7 = = 0. x (5/) /6 x = x (5/) = = + = x (5/) 0 7/6 ad so o... FIGURE 8 hs algorthm s ot mportat curretly as a optmsato tool: t s too slow to coverge. We use t to motvate more complcated algorthms. Exercse: Cosder the followg learly costraed covex programmg problem. 3 Ÿ m f(x, x ) = - 3 x - 4 x + (x ) + (x ) x + x ; x, x 0. Start from (/4, /4) ad apply three teratos of the Frak-Wolfe Algorthm.. Use KK codtos to check f the soluto s correct. 3. Program the Frak-Wolfe algorthm to solve the problem. (here are may publc doma LP solvers you ca use. For example, there s o Numercal recpes C.) 0. BARRIER FUNCIONS FOR GENERAL CONSRAINED PROBLEMS I covex programmg, ay local mmum (maxmum) s also a global mmum (maxmum). I geeral NLP that arse practce do ot exactly satsfy covexty assumptos. A commo approach to such problems s to apply a algorthmc search procedure that wll stop whe t fds a local mmum (or maxmum) ad the to restart t from a umber of tmes from a varety of tal tral solutos order to fd as may dstct

29 - (00) Itroducto 3 - local mma (maxma) as possble. he best of these optma s the chose as the decso to mplemet. Normally, the search procedure s oe that has bee desged to fd a global mmum (maxmum) whe all the assumptos of covex programmg hold, but t ca also operate to fd a local mmum (maxmum) whe they do ot. Oe such procedure that has bee wdely used s the sequetal ucostraed mmsato techque (SUM) due to Facco & McCormck. here are actually two versos of SUM, oe of whch s a exteror pot algorthm that starts wth feasble solutos whle usg a pealty fucto to force covergece to the feasble rego. Pealty fuctos are dscussed later ths course. We ow descrbe the other verso, s a teror pot algorthm that deals drectly wth feasble solutos whle usg a barrer fucto to force the soluto to rema sde the feasble rego. SUM replaces the orgal problem by a sequece of ucostraed optmsato problems whose soluto coverge to a soluto (local mmum) of the orgal problem. he ucostraed optmsato algorthms covered ths course are the steepest descet method dscussed Secto 7 ad the Goldste-Levt-Polyak algorthm for covex programmg, whch, the absece of costrats, reduces mmedately to Newto or quas-newto algorthms for ucostraed optmsato. Each of the ucostraed problems the SUM sequece volves choosg a successvely smaller strctly postve value of a scalar ( ad the solvg m x m x j (x, () = f (x) ( B (x). B (x) s a barrer fucto that has the followg propertes (for x that are feasble for the orgal problem):. B (x) s small whe x s far from the boudary of the feasble rego,. B (x) s large whe x s close to the boudary of the feasble rego, 3. B (x) Ä as dstace of x to earest boudary of the feasble rego Ä 0. As h (x) Ÿ 0, examples of B (x) are = boud costrats x 0, j /x (Caroll, 96), j= h (x) log h (x) = (Frsch, 955). hus, by startg the search procedure wth a teror feasble tal pot (h (x) 0) ad the attemptg to decrease j (x, (), B (x) provdes a barrer that prevets the search from ever crossg (or eve reachg) the boudary of the feasble rego for the orgal problem. Note that, for feasble values of x, the deomator of each term j = x j

30 - (00) Itroducto 4 - s proportoal to the dstace of x from the boudary of the oegatvty costrat. Cosequetly, each term s a boudary repulso term that has all the precedg three propertes wrt the partcular costrat boudary. Aother attractve feature of B (x) s that, whe all the assumptos of covex programmg are satsfed, j (x, () s a covex fucto of x. What happes f the soluto s at the boudary? hs s the reaso SUM volves a sequece of these ucostraed optmsato problems for successvely smaller values of ( approachg zero. he soluto of each ucostraed optmsato problem becomes the tal pot for the ext. For example, each ew ( may be obtaed from the precedg oe by multplyg wth a costat ), 0 ), where typcal values are ) = 0. or 0.0. As ( Ä 0, j (x, () approaches f (x), so the correspodg local mmum coverges to a local mmum of the orgal problem. herefore, t s ecessary to solve oly eough ucostraed optmsato problems to permt extrapolatg ther solutos to ths lmtg soluto. Example 8: " # m x + x 4 x 4 x + (x ) + (x ) + 7 Ÿ 0 hus the feasble rego s the dsc wth radus, cetred at,, that s: (x ) + (x ) Ÿ. he soluto s (, ) wth objectve fucto value x + x = 4. m x m x " # j(x, () = x + x ( log 4 x + 4 x (x ) (x ) 7 he ecessary codtos for a ucostraed optmum are `j(x, () `j(x, () From these equatos, we see " # = ( (4 x ) 4 x + 4 x (x ) (x ) 7 = 0 " # = ( (4 x ) 4 x + 4 x (x ) (x ) 7 = 0 ad solvg oe of these equatos, we have x( () = x( () ( x (() = (

31 - (00) Itroducto 5 - where the egatve secod term correspods to the mmum. he soluto of the problem s approached by a sequece of x ((), as ( Ä 0. We have, therefore, lm ( x = x = ( Ä0 ( + 8 = =.989 Let us suppose that we start the process of mmsato wth ( = 0. Usg a sutable mmsato techque, we should be able to fd the soluto x (0) =.9504, We decrease ( by a factor of 0 ad restart the optmsato wth ( = ad the tal values for x just computed. By cotug the process, the trajectory of mma s geerated for decreasg values of (. We see that for ( small eough, say ( = 0.000, the method produces qute a good approxmato to the soluto. ( x( () = x( () ã ã he SUM algorthm Step 0: Gve tal feasble pot x 0 ot o the boudary, set ( 0 0, ) (0, ), k = 0. Step : Usg x k as the startg pot solve the followg usg a ucostraed optmsato method, to fd a local mmum m k+ k x k k x = x (( ) = arg j (x, ( ) = f (x) ( B (x). Step : x x x k+ k If Ÿ macheps, stop. We have reached a local optmum. Else, set k ( k+ = ) ( k, k = k + ad go to Step. Exercse:. Solve the exercse at the ed of secto 9 wth the gve tal pot usg SUM by - -4 applyg the sequece ( =, 0, 0.. Program SUM to solve ths problem. ( 0 ad ) are data that are set by the user. Use a ucostraed optmsato subroute from NAG lbrary. 3. How would we adapt SUM to a maxmsato problem?