Optimization. Nuno Vasconcelos ECE Department, UCSD

Transcription

1 Optmzaton Nuno Vasconcelos ECE Department, UCSD

2 Optmzaton many engneerng problems bol on to optmzaton goal: n mamum or mnmum o a uncton Denton: gven unctons, g,,...,k an h,,...m ene on some oman Ω R n mn subject to, Ω g, h, : cost; h equalty, g nequalty: constrants or compactness e rte g nstea o g,. Smlarly h note that no nee or

3 Optmzaton note: mamzng s the same as mnmzng, ths enton also orks or mamzaton the easble regon s the regon here. s ene an all constrants hol R { Ω g, h } * s a global mnmum o *, Ω * s a local mnmum o ε > s.t. * < ε * local global 3

4 he graent the graent o a uncton at z s h th t t n z z z,, L heorem: the graent ponts n the recton o mamum groth proo: proo: rom aylor seres epanson α α α O ervatve along,.cos. lm α α α * 4 s mamum hen s n the recton o the graent

5 he graent note that there s no recton o groth also -, an there s no recton o ecrease e are ether at a local mnmum or mamum or sale pont conversely, at local mn or ma or sale pont no recton o groth or ecrease ths shos that e have a crtcal pont an only to etermne hch type e nee secon orer contons ma mn sale 5

6 ma he Hessan, by aylor seres α an 3 α α α α O 3 an α α α O mn pck α such that Oα <<, mamum at an only, mnmum at an only sale mnmum at an only, sale otherse ths proves the ollong theorems 6 ths proves the ollong theorems

7 Mnma contons unconstrane let be contnuously erentable * s a local mnmum o an only s a local mnmum o an only has zero graent at * * an the Hessan o at * s postve ente n t R * here R, n M L 7 n n L

8 Mama contons unconstrane let be contnuously erentable * s a local mamum o an only s a local mamum o an only has zero graent at * * an the Hessan o at * s negatve ente n t R * here R, n M L 8 n n L

9 Eample conser the unctons g g the graents are has no mnma or mama, g g has a crtcal pont at the orgn, snce Hessan s postve ente, ths s a mnmum g 9

10 Eample makes sense because s a plane, graent s constant so-contours o -

11 Eample makes sense because g g s a quaratc, postve everyhere but the orgn note ho graent ponts toars largest ncrease g p g g h h h g h

12 Conve unctons Denton: s conve,u Ωan λ [,] λ λ u λ λ u heorem: s conve an only ts Hessan s postve ente t or all t, Ω proo: λ-λv u requres some ntermeate results that e ll not cover λ-λv u e ll skp t λ-λv

13 Concave unctons Denton: s concave,u Ωan λ [,] λ λ u λ λ u heorem: s concave an only ts Hessan s negatve ente t or all t, Ω proo: - s conve by prevous theorem, Hessan s negatve ente Hessan o s postve ente 3

14 Conve unctons heorem: s conve any local mnmum * s also a global mnmum Proo: e nee to sho that, or any u, * u or any u: *-[λ*-λu -λ *-u an, makng λ arbtrarly close to, e can make *-[λ*-λu ε, or any ε > snce * s local mnmum, t ollos that * λ*-λu an, by convety, that * λ*-λu λu or *-λ u-λ an * u 4

15 Constrane optmzaton n summary: e kno hat are contons or unconstrane ma an mn e lke conve unctons n a mnma, t ll be global mnmum hat about optmzaton th constrants? a e entons to start th nequalty g : s actve g, otherse nactve nequaltes can be epresse as equaltes by ntroucton o slack varables g g ξ, an ξ 5

16 Conve optmzaton Denton: a set Ω s conve,u Ωan λ [,] then λ-λu Ω a lne beteen any to ponts n Ω s also n Ω conve not conve Denton: an optmzaton problem here the set Ω, the cost an all constrants g an h are conve s sa to be conve note: lnear constrants g Ab are alays conve zero Hessan 6

17 Constrane optmzaton e ll conser general not only conve constrane optmzaton problems, start by case th only equaltes heorem: conser the problem * arg mn subject to h here the constrant graents h * are lnearly nepenent. hen * s a soluton an only there ets a unque vector λ, such that m λ * λ h * y * m λ h * y, y s.t. h * y 7

18 Alternatve ormulaton state the contons through the Lagrangan L, λ m λ h the theorem can be compactly rtten as * L *, λ y λ L *, * λ * L *, λ y, y s.t. h * y the entres o λ are reerre to as Lagrange multplers 8

19 Graent revste recall rom * that ervatve o along s lm α α α..cos, ths means thatt greatest ncrease hen no ncrease hen snce there s no ncrease hen s tangent to so-contour k the graent s perpencular to the tangent o the so-contour ths suggests a geometrc proo no ncrease 9

20 Lagrangan optmzaton geometrc nterpretaton: snce h s a so-contour o h, h* s perpencular to the so-contour says that * span{h *}.e. to tangent space o the constrant surace ntutve recton o largest ncrease o s to constrant surace the graent s zero along the constrant no ay to gve an nntesmal graent step, thout enng up volatng t t s mpossble to ncrease an stll satsy the constrant h span{h*} tg plane

21 Eample conser the problem mn subject to t leas to the ollong pcture h so-contours o h -

22 Eample conser the problem mn subject to to the so-contours o k h h -

23 Eample conser the problem mn subject to h to the so-contour o h - h h - 3

24 Eample recall that ervatve along s α lm..cos, α α crtcal pont crtcal pont - movng along the tangent s escent as long as cos tg, < -.e. π/ < angle,tg < 3π/ - can alays n such unless tg - crtcal pont hen h - to n hch type e nee n orer as beore 4

25 Alternatve ve conser the tangent space to the so-contour h ths s the subspace o rst orer easble varatons { h * } V *, space o or hch satses the constrant up to rst orer appromaton V* easble varatons h * h* 5

26 Feasble varatons multplyng our rst Lagrangan conton by * λ h * t ollos that m *, V * ths s a generalzaton o * n unconstrane case mples that * V* an thereore * h* note: Hessan constrant only ene or y n V* makes sense: e cannot move anyhere else, oes not really matter hat Hessan s outse V* 6

27 In summary or a constrane optmzaton problem, th equalty constrants heorem: conser the problem * arg mn subject to h here the constrant graents h * are lnearly nepenent. hen * s a soluton an only there ets a unque vector λ, such that m λ * λ h * y * m λ h * y, y s.t. h * y 7

28 Alternatve ormulaton state the contons through the Lagrangan L, λ m λ h the theorem can be compactly rtten as * L *, λ y λ L *, * λ * L *, λ y, y s.t. h * y the entres o λ are reerre to as Lagrange multplers 8

29 9