Prosemnar Optmerung II Vctor A. Kovtunenko Insttute for Mathematcs and Scentfc Computng, Karl-Franzens Unversty of Graz, Henrchstr. 36, 8010 Graz, Austra; Lavrent ev Insttute of Hydrodynamcs, Sberan Dvson of the Russan Academy of Scences, 630090 Novosbrsk, Russa SS 01/013: LV 61.35 1
V.A. Kovtunenko: Prosemnar Optmerung II Список литературы [1] D.P. Bertsekas, A. Nedć, A.E. Ozdaglar, Convex Analyss and Optmzaton. Athena Scentfc, Belmont, 003, 534 pp. [] C. Geger, C. Kanzow, Numersche Verfahren zur Lösung unrestrngerter Optmerungsaufgaben. Sprnger-Verlag, Berln, 1999, 487 S. [3] C. T. Kelley, Iteratve Methods for Optmzaton. SIAM, Phladelpha, PA, 1999, 180 pp. [4] A.M. Khludnev, V.A. Kovtunenko, Analyss of Cracks n Solds. WIT-Press, Southampton, Boston, 000, 408 pp. [5] D.G. Luenberger, Y. Ye, Lnear and Nonlnear Programmng. Sprnger, New York, 008, 546 pp. [6] J. Nocedal, S.J. Wrght, Numercal Optmzaton. Sprnger-Verlag, New York, 1999, 636 pp. [7] В.М. Алексеев, Э.М. Галеев, В.М. Тихомиров, Сборник задач по оптимизации. Москва: Наука, 1984, 88 с.
(1.1) V.A. Kovtunenko: Prosemnar Optmerung II 3 1. Lnear programs Lnear programs (LP) are stated n the canoncal (standard) form: mn c x over x R n subject to Ax = b, x 0, where c R n, b R m, A R m n, and m < n. The feasble set s defned (1.) F = {x R n : Ax = b, x 0}, and ts boundary corresponds to the actve set (1.3) A = {x R n : Ax = b, x j = 0}. The fundamental theorem of LP states that optmal solutons of (1.1) occur along A at extreme (corner) ponts of F. Exercse 1.1. Rewrte the followng LP n the standard form: mn{x 1 + x + 3x 3 } over x R 3 subject to 4 x 1 + x 5, 6 x 1 + x 3 7, x 1 0, x 0, x 3 0, and determne A, b, c n (1.1). Hnt: ntroduce slack varables x 4 0, x 5 0, x 6 0, x 7 0 to satsfy the non-standard nequalty constrants. Exercse 1.. Rewrte the followng LP n the standard form: mn{x + y + z} x + y 1, x + z = 3, over (x, y, z) R 3 subject to and determne A, b, c n (1.1). Hnt: ntroduce non-negatve varables (x 1,..., x 6 ) by x = x 1 x, y = x 3 x 4, z = x 5 x 6. Exercse 1.3. Convert the l 1 -mnmzaton problem to LP: mn x over x R 1 subject to Ax = b. Hnt: use the nequalty x x x. Exercse 1.4. Gven the constrants: x 1 + x 16, x 1 + x 1, x 1 + x, x 1 0, x 0, a) plot the feasble set F and determne the extreme (corner) ponts, b) plot the actve set A. Exercse 1.5. Consder feasble set F R satsfyng the followng nequaltes: x 1 0, x 0, x x 1, x 1 + x 7. a) Plot F and lst ts extreme ponts, b) compute mn{x 1 x } over x R subject to x F, c) compute max{x 1 x } over x R subject to x F. Hnt: apply the fundamental theorem of LP.
4 V.A. Kovtunenko: Prosemnar Optmerung II (.1). Nonlnear programs We consder nonlnear programs (NLP) of the general form: mn J(x) over x R n subject to e(x) = 0, g(x) 0, where J : R n R, e : R n R m, e = (e 1,..., e m ), m < n, and g : R n R p, g = (g 1,..., g p ). The feasble set s defned (.) F = {x R n : e(x) = 0, g(x) 0}, and ts boundary corresponds to the actve set (.3) A = {x R n : e(x) = 0, g j (x) = 0}. A vector x F s called a global soluton of (.1) f J(x ) J(x) for all x F, respectvely, a local soluton f there s a neghborhood U(x ) F such that J(x ) J(x) for all x U(x ). Exercse.1. In R consder the constrants: x 1 0, x 0, x (x 1 1) 0, x 1. Plot the feasble set F n (.) and show that x = (1, 0) s feasble but not regular. Hnt: A pont x at the hyperplane E = {x R n : e 1 (x) = 0,..., e k (x) = 0} s regular f vectors e 1 (x ),..., e k (x ) are lnearly ndependent (called lnear ndependence constraned qualfcaton (LICQ)). Exercse.. Let F be a convex set n R. Ths mples that for all x, y F and t [0, 1] ponts tx + (1 t)y F. Prove that, for nonnegatve weghts c 1,..., c k R such that k =1 c = 1, f x 1,..., x k F then k =1 c x F. Hnt: use nducton over k. Exercse.3. Determne, whether the followng functons defned on R + a) J(x) = x 1 x b) J(x) = 1 x c) J(x) = x 1 (ln x 1 1) + x (ln x 1) 1 x are convex, concave, or nether. Hnt: twce dfferentable functon J s convex on a convex set f ts Hessan matrx D( J) s postve semdefnte (spd). Exercse.4. If the objectve functon J s convex on the convex feasble set F, prove that local solutons of (.1) are also global solutons. Exercse.5. Consder the problem: mn{x 1 + x } subject to x 1 + x = 1. a) Solve the problem by elmnatng the varable x. b) Plot F and fnd tangent plane at feasble pont x = ( 1, 1 ). Hnt: at a regular pont x of hyperplane E the tangent plane (tangent cone) s (.4) T (x ) = {v R n : De(x ) v = 0} wth De := ( e 1,..., e k ). c) Fnd (the Lagrange multpler) λ R such that J(x ) = De(x )λ. d) Illustrate ths problem geometrcally.
(3.1) V.A. Kovtunenko: Prosemnar Optmerung II 5 3. Equalty constraned optmzaton For the equalty constraned mnmzaton problem mn J(x) over x R n subject to {e 1 (x) = 0,..., e m (x) = 0} the Lagrange functon (Lagrangan) s defned by (3.) L : R n+m R, L(x, λ) = J(x) + e(x) λ wth the Lagrange multplers (dual varables) λ R m. The frst order necessary condton of optmalty mples a statonary pont (x, λ ) solvng (3.3) (x,λ) L(x, λ ) = 0 { J(x ) + De(x )λ = 0, e(x ) = 0}, and the second order suffcent condton along the tangent plane T (x ) reads (3.4) v D( L)(x, λ ) v > 0 for all v T (x ), where the Hessan matrx (Hessan) D( L) : R n+m R n n. Exercse 3.1. Solve usng Lagrange multplers: mn{ x 3 y 3 } subject to x + y = 1. Exercse 3.. Consder the followng mnmzaton problem: x 1 1 x + 1 x 3 + x 1 x } subject to x1 + x = 4, x 3 9x 1 + 16 = 0. a) Construct Lagrangan and fnd the statonary pont (x, λ ) of ths problem. b) Check a second order suffcent condton along the tangent plane T (x ). Exercse 3.3. If a, b, c are postve real numbers, prove that the nequalty a 3 + b 3 + c 3 3abc holds. Hnt: t s enough to prove that, f 3abc = K (wth arbtrarly fxed K 0), then mn{a 3 + b 3 + c 3 } = K. Exercse 3.4. A box wth sdes x, y and z s to be manufactured such that ts top (xy), bottom (xy), and front (xz) faces must be doubled. Fnd the dmensons of such a box that maxmze the volume xyz for a gven face area, equal to 18 dm. Hnt: use frst order and verfy second order condtons. Exercse 3.5. Fnd an orented trangle of maxmal area such that one vertex s (x 3, y 3 ) = (1, 0) and the other two vertexes (x 1, y 1 ), (x, y ) le on the unt crcle n R. Hnt: use the area formula S = 1 x 1 x x 3 y 1 y y 3 1 1 1.
6 V.A. Kovtunenko: Prosemnar Optmerung II (4.1) 4. Inequalty constraned optmzaton For the general, equalty and nequalty constraned mnmzaton problem mn J(x) over x R n subject to e(x) = 0, g(x) 0, the Lagrange functon (Lagrangan) s defned by (4.) L : R n+m+p R, L(x, λ, µ) = J(x) + e(x) λ + g(x) µ wth the Lagrange multplers (λ, µ) R m+p. The frst order necessary optmalty condton mples Karush Kuhn Tucker (KKT) condtons: { (x,λ) L(x, λ, µ J(x ) = 0 ) + De(x )λ + Dg(x )µ = 0, (4.3a) e(x ) = 0 (4.3b) µ 0, g(x ) 0, g (x )µ = 0 for = 1,..., p. The second order suffcent condton along the tangent plane T (x ) reads (4.4) v D( L)(x, λ, µ ) v > 0 for all v T (x ). Exercse 4.1. Solve usng KKT condtons: mn{3x x 3 } over x R subject to x and plot the objectve functon on the feasble set. Exercse 4.. Solve usng KKT condtons: mn{e x 1 } over x R subject to e x 1 + x = 7, x 1 0, x 0. Exercse 4.3. Verfy that complementarty condtons (4.3b) follows from the varatonal nequalty (4.5) µ 0, µ L(x, λ, µ ) (µ µ ) 0 for all {µ R p : µ 0}. Hnt: plug µ = (µ 1,..., tµ,..., µ p) wth arbtrary t > 0 and {1,..., p}. Exercse 4.4. Let x, λ, µ 0 be a soluton of the mnmax problem: (4.6) mn max L(x, λ, µ) over (x, λ, µ) (x,λ) µ Rn+m+p subject to µ 0. (a) Verfy that ths soluton satsfy KKT condtons (4.3). (b) From (4.6) derve that x s feasble and that t solves (4.1). Exercse 4.5. Consder the followng mnmzaton problem: mn{ (x ) (y 1) } subject to x + 4y 3 and x y. Solve ths problem usng KKT condtons.
V.A. Kovtunenko: Prosemnar Optmerung II 7 5. Regularty and senstvty Exercse 5.1. Consder the mnmzaton problem: mn{ x 1 } over x R subject to x 1 x and x 0. Fnd the global soluton of ths problem and verfy that there s no KKT ponts. Hnt: consder the feasble set and check ts regularty. Exercse 5.. For the problem: mn{y x} over R subject to y (1 x) 3, x + y 1, y 0, verfy that the objectve functon s convex, the feasble set s convex and has a non-empty nteror (Slater condton). Ths wll guarantee that the KKT pont s the soluton of the convex mnmzaton problem. Fnd ths soluton. Check a second order suffcent condton as well. Exercse 5.3. Consder the problem: mn{4x 1 + x } over x R subject to x 1 + x 9, x 1 0, x 0. (a) Verfy that the feasble set F s non-convex (hence, a KKT pont may be not the soluton of the mnmzaton problem). (b) Snce the objectve functon s lnear, fnd the global soluton of the problem at the boundary (the actve set A) of F. (c) Fnd all solutons of the KKT system at A and determne ts knd of extrema. Exercse 5.4. Consder the followng mnmzaton problem: αx 1 + 1 x + 5x 1 } over x R subject to x 1 0. Usng frst and second order optmalty condtons fnd ts soluton n dependence of the parameter α R. Exercse 5.5. Consder the problem: mn { (x 1) + y } over R subject to x 1 β y. For what values of the parameter β R ts KKT pont s a local soluton? Hnt: use a second order suffcent condton.
8 V.A. Kovtunenko: Prosemnar Optmerung II 6. Quadratc programs We consder quadratc programs (QP) as NLP wth the specfc (quadratc) objectve functon J(x) = 1 x Qx + d x, where the matrx Q R n n s symmetrc postve defnte (spd) and d R n. Exercse 6.1. Verfy that, for Q spd(r n n ), the quadratc objectve functon J s convex by the defnton: J ( tx+(1 t)y ) tj(x)+(1 t)j(y). Hnt: use the Cauchy Schwarz nequalty x Qy 1 x Qx + 1 y Qy. Exercse 6.. Consder the quadratc mnmzaton problem: (6.1) x Qx + d x } subject to Ax = b, where A R m n and b R m. Prove that x s a local soluton of the problem f and only f t s a global soluton. Exercse 6.3. The problem of fndng the shortest dstance from a pont x 0 R n to the hyperplane {x R n : Ax = b} can be formulated as (x x0 ) (x x 0 ) } subject to Ax = b. (a) Verfy that the problem s of the form (6.1) and determne Q and d. (b) Show that: the matrx AA s nonsngular f A has full rank, (c) the statonary pont s x = x 0 A λ and λ = (AA ) 1 (Ax 0 b), (d) the statonary pont s the optmal soluton. Exercse 6.4. For gven data ponts (x 1, y 1 ),..., (x n, y n ) n R, the lnear regresson problem conssts n fttng the lne y = d+kx such that to mnmze the resduals: n } mn{ 1 (d + kx y ) (d + kx y ) over (d, k) R. =1 (a) Rewrte the problem n the followng form of unconstraned QP: (Az b) (Az b) } over z R. (b) Usng optmalty derve the normal equatons: A Az = A b. (c) Solvng these two equatons verfy the formula: k = Sxy xȳ S xx x, d = ȳ k x wrtten n statstcal terms S xx = 1 n n =1 x, S xy = 1 n n =1 x y, x = 1 n n =1 x, ȳ = 1 n n =1 y. Exercse 6.5. Consder the problem of fndng the pont on the parabola y = 1 5 (x 1) that s closest to (x 0, y 0 ) = (1, ). Ths can be formulated as (x 1) + 1 (y 1)} subject to (x 1) = 5y. Fnd the statonary pont of the problem and show that t s the mnmum pont usng a second order condton.
V.A. Kovtunenko: Prosemnar Optmerung II 9 7. Complementarty Exercse 7.1. The Eucldean projecton P S x 0 of a pont x 0 R n on smplex S = {x R n : x 0, x e = 1} ( e = (1,..., 1) ) solves the problem: x x0 } over x R n subject to x S. Verfy the formula P S x 0 = max(0, x 0 λ e), where λ R s a Lagrange multpler assocated to the constrant x e = 1. Exercse 7.. For the maxmzaton of entropy fnd the soluton: max { ( x 1 log(x 1 ) + + x n log(x n ) )} over x R n subject to x S. Exercse 7.3. φ : R R s called a nonlnear complementarty problem (NCP) functon f t satsfes φ(x, y) = 0 x 0, y 0, x y = 0. Verfy that the followng are NCP functons: φ mn (x, y) = mn(x, y), φ max (x, y) = y max(0, y cx) wth c > 0, φ FB (x, y) = x + y (x + y) (Fscher Burmester) Exercse 7.4. Consder an abstract nequalty constraned mnmzaton (7.1) mn J(x) over x R n subject to Ax = b, x 0. (a) Usng KKT condtons and a NCP functon φ derve the NCP Φ(x, λ, µ ) := J(x ) + A λ µ (7.) Ax b = 0. φ(x, µ ) (b) If there exsts ndex such that both x = µ = 0, then prove that the matrx of dervatves D (x,λ,µ) Φ(x, λ, µ ) s sngular. (c) Usng φ max rewrte φ max (x, µ ) = 0 as the lnear system x = 0 for A(x, µ ), µ = 0 for I(x, µ ) wth respect to prmal-dual strctly actve A and nactve I sets of ndexes A(x, µ ) = { : µ cx > 0 }, I(x, µ ) = { : µ cx 0 }. (d) On ths bass suggest teratons solvng nonlnear problem (7.). Exercse 7.5. Consder the nequalty constraned optmzaton mn 1 { x 1 + (x ) + (x 3 + 3) } subject to x 1 0, x 0, x 3 0. Intalzng A ( 1) =, I ( 1) = {1,, 3} terate the actve and nactve sets A (k) = { : µ (k) cx (k) > 0 }, I (k) = { : µ (k) cx (k) 0 } together wth x (k) = 0 for A (k 1) and µ (k) = 0 for I (k 1). From the teraton fnd a soluton of the KKT system for ths problem.
10 V.A. Kovtunenko: Prosemnar Optmerung II 8. Approxmatons Exercse 8.1. Let x = argmn{j(x)} over x R n subject to e(x) = 0, g(x) 0. Verfy that there exsts constant c > 0 such that x s also a mnmzer of the penalty problem mn { J(x) + c ( e(x) 1 + max(0, g(x)) 1 )} over x R n. Hnt: use the fact that the assocated Lagrangan L satsfes L(x, λ, µ ) L(x, λ, µ ) for all x R n. Exercse 8.. (a) For the constraned mnmzaton problem mn { J(x) = (x 1 6) + (x 7) } over x R subject to x 1 + x 7 fnd the soluton x and the assocated Lagrange multpler λ. (b) Consder the penalty problem mn { J α (x) = J(x) + α ( max(0, x 1 + x 7) ) } over x R and fnd ts soluton x α n dependence of the parameter α > 0. Is x α nteror or exteror to the feasble set? (c) Verfy that x α x and α max(0, x α 1 + xα 7) λ as α. Exercse 8.3. (a) Fnd a soluton x of the constraned mnmzaton problem mn { J(x) = x 1 + 9x } over x R subject to x 1 + x 4. τ x 1 +x 4 (b) Consder the assocated nverse barrer functon J τ (x) = J(x) + n dependence of the parameter τ > 0. Fnd the soluton x τ of the unconstraned problem: mn{j τ (x)} over x R, and compare x τ wth x. Exercse 8.4. For τ > 0 consder a log-barrer functon constraned mnmzaton mn { n (8.1) J(x) τ log(x ) } over x R n subject to Ax = b, x > 0. =1 (a) Verfy that ts KKT condtons lead to the nteror pont system J(x) + A λ ν = 0 (8.) Ax b = 0. x ν = τ for = 1,..., n, x > 0, ν > 0. (b) Verfy that, conversely, f the nteror pont system (8.) s solvable, then t follows the KKT system for (8.1). Exercse 8.5. For the nequalty constraned optmzaton problem mn { J(x) = 5x 1 + x } over x R subject to x 1 1, x 0, wrte the nteror pont system and fnd ts soluton n dependence of parameter τ > 0. Passng τ 0 fnd a soluton of KKT system for ths problem.
V.A. Kovtunenko: Prosemnar Optmerung II 11 9. Smplex method For LP: mn{c x} over x R n subject to Ax = b, x 0 wth A R m n, parttonng A = (B, D), x = (x B, x D ), c = (c B, c D ) mples the system Ix b + B 1 Dx D = B 1 b, (c D c BB 1 D)x D = c x c BB 1 b expressed by Tableau: ( I B 1 D B 1 ) b T := 0 c D c B B 1 D =: r c B B 1 b and follows the smplex algorthm: Repeat untl r 0: select j = argmn {r k : r k < 0}, = argmn k {1,...,n} k {1,...,m} { (B 1 b) k (B 1 D) kj : (B 1 b) k (B 1 D) kj 0 }, pvot on (B 1 D) j. If all (B 1 b) k (B 1 D) kj < 0 then the problem s unbounded. Exercse 9.1. Usng the smplex algorthm, terate x = B 1 b from T for max{5x 1 + x } over x R subject to 4x 1 + 3x 1, x 1 + 3x 6, x 1 0, x 0. Sketch the feasble set and the path of the smplex steps stoppng at the soluton of ths LP. Are the teratons of x feasble? Exercse 9.. Usng the smplex algorthm, solve the followng LP: max{x 1 + x } over x R subject to x 1 + x 1, x 1 x 1, x 1 0, x 0. Sketch the feasble set and the path of the smplex steps. Exercse 9.3. Fnd all solutons of the LP: max{x 1 + x + x 3 } over x R 3 subject to x 1 + x + x 3, x 1 + 4x + x 3 4, x 1 0,..., x 3 0. Exercse 9.4. Fnd an optmal soluton of the problem: mn { x 1 3x 5 x 3} over x R 3 subject to x 1 0,..., x 3 0, 3x 1 x + x 3 7, x 1 + 4x 1, 4x 1 + 3x + 3x 3 14. Exercse 9.5. Consder the followng LP: mn{5x 1 + 3x } over x R 3 subject to x 1 0,..., x 3 0, x 1 x + 4x 3 4, x 1 + x + x 3 5, x 1 x + x 3 1. Startng pvot element (3, 3), solve the problem wth the (dual) smplex algorthm.