e Karus-Kun-ucker condtons and dualt Nuno Vasconcelos ECE Department, UCSD
Optmzaton goal: nd mamum or mnmum o a uncton Denton: gven unctons, g, 1,...,k and, 1,...m dened on some doman Ω R n mn w, w Ω subect to g w w,, or compactness we wrte gw nstead o g w,. Smlarl w we derved necessar and sucent or local optmalt n te absence o constrants equalt constrants onl
Mnma condtons unconstraned let w be contnuousl derentable w s a local mnmum o w and onl w s a local mnmum o w and onl as zero gradent at w w and te Hessan o at w s postve dente w n t d d w d R were d d w d R, 1 n M L 3 1 1 n n L
Mama condtons unconstraned let w be contnuousl derentable w s a local mamum o w and onl w s a local mamum o w and onl as zero gradent at w w and te Hessan o at w s negatve dente w n t d d w d R were d d w d R, 1 n M L 4 1 1 n n L
Constraned optmzaton wt equalt constrants onl eorem: consder te problem arg mn subect to were te constrant gradents are lnearl ndependent. en s a soluton and onl tere ets a unque vector λ, suc tat m λ 1 + λ + m 1 λ, s.t. 5
Alternatve ormulaton state te condtons troug te Lagrangan m t t b tl tt, 1 L m + λ λ te teorem can be compactl wrtten as, L λ,, L L L st L, λ λ λ λ λ te entres o λ are reerred to as Lagrange multplers,, L s.t. λ 6
Geometrc nterpretaton dervatve o along d s lm α w + αd α w d w d. w.cos d, w ts means tatt greatest ncrease wen d no ncrease wen d snce tere s no ncrease wen d s tangent to so-contour k te gradent s perpendcular to te tangent o te so-contour allows geometrc nterpretaton o te Lagrangan g condtons no ncrease 7
Lagrangan optmzaton geometrc nterpretaton: snce s a so-contour o, s perpendcular to te so-contour sas tat span{ }.e. to tangent space o te constrant surace ntutve drecton o largest ncrease o s to constrant surace te gradent s zero along te constrant no wa to gve an nntesmal gradent step, wtout endng up volatng t t s mpossble to ncrease and stll sats te constrant span{} tg plane 8
Eample consder te problem mn 1 + subect to 1 + 1 1 to te so-contours o 1 + k 1 1 1 1 + 1 + 1 9 1 + -1
Eample consder te problem mn 1 + subect to 1 + to te so-contour o 1 + - 1 + 1 1 1 1 1 + 1 + -1 1
Eample recall tat dervatve along d s w + +α d w lm d. w.cos d, w α α crtcal pont crtcal pont - movng along te tangent s descent as long as cos tg, < -.e. π/ < angle,tg < 3π/ - can alwas nd suc d unless tg - crtcal pont wen - to nd wc tpe we need nd order as beore 11
Alternatve vew consder te tangent space to te so-contour ts s te subspace o rst order easble varatons { } V, space o or wc + satses te constrant up to rst order appromaton V easble varatons 1
Feasble varatons multplng our rst Lagrangan condton b + λ t ollows tat m 1, V ts s a generalzaton o n unconstraned case mples tat V and tereore note: Hessan constrant onl dened or n V makes sense: we cannot move anwere else, does not reall matter wat Hessan s outsde V 13
Inequalt constrants wat appens wen we ntroduce nequaltes? arg mn subect to, g we start b denng te set o actve nequalt constrants { g } A or eample 1, 1 +, -5 1 5, -5 5 14
Actve nequalt constrants we ave a mnmum at,-5 1 5 <, - 1-5 <, and 5 < arenactve - 5 s actve -5 note tat a local mnmum or ts problem would stll be a local mnmum we removed te nnactve constrants nnactve constrants do not do antng actve constrants are equaltes nnactve actve 15
Constraned optmzaton ence, te problem arg mn subect to, g s equvalent to arg mn subect to, g, A ts s a problem wt equalt constrants, tere must be a λ and µ, A, suc tat + m 1 λ + µ g A wc does not cange we assgn a zero Lagrange g multpler to te nnactve constrants 16
Constraned optmzaton lettng µ, A, + m 1 r λ + µ g 1 tere s one nal constrant t wc s µ, or all due to te ollowng pcture g as to pont nward oterwse we would ave a mamum o g as to pont outward oterwse g would ncrease nward,.e. g would be non-negatve g nsde wen we put togeter all tese constrants we obtan te amed Karus-Kun-ucker Kun KK condtons 17
e KK condtons eorem: or te problem, arg mn g to subect s a local mnmum and onl tere est λ and µ suc tat 1 1 g r m + + µ λ,, v A r m µ µ { },,, 1 1 A g V were V g v r m + + and µ λ 18 { },, A g V were and
Geometrc nterpretaton Let s orget te equalt constrants or now later we wll see tat te do not cange muc consder te problem arg mn subect to g rom te KK condtons, te soluton satses µ [ L, µ ], µ, A wt L, µ r + 1 µ g 19
Geometrc nterpretaton wc s equvalent to [ ] g [ L, µ ] mn + µ L mn µ wt we tus ave µ, and µ, A + µ g L + µ g L or w z - b w z - b were 1 b L, w, z µ g plane n z-space normal w, bas b L s n al-space ponted to b w
Geometrc Interpretaton rom b we ave L, 1 w, z µ g snce µ, w s alwas n te rst quadrant snce rst coordnate s 1, w s never parallel to g as ts can be vsualzed n z-space space as also, two cases: 1 g w z - b L te -ntercept s,l, s te mnmum o L w admssble planes w R g,, g R r-1 1
Geometrc Interpretaton b L, 1 w, µ case g< z g te constrants are nnactve µ w 1, plane s orzontal w z - b L te -ntercept s,l, s te mnmum o L.e. ts s alwas te case n general m o actve and nnactve but beavor s ts w g, g< R, g R r-1
In summar L mn wt µ [ g ] [ L, µ ] mn + µ s equvalent to, and µ, A w z - b 1 b L, w w, z z - b µ g can be vsualzed as g R g< w R easble g,, w g,, g R r-1 g< g R r-1 3
Dualt [ g ] [ L, µ ] mn + L mn µ wt µ, and µ, A does not appear terrbl dcult once I know µ ow I do nd ts value? consder ts uncton or an µ [ L, µ ] mn[ g ] q µ mn + µ wt µ ts s equvalent to w z - b b q µ, w w z - b 1, z µ g te pcture s te same wt L replaced b qµ 4
Dualt notng tat perplane w,b stll as to support easble set and we stll ave µ> ts leads to w g R g< w easble w R g,, w,,qµ g R r-1 g< g R r-1,qµ 5
Dualt note tat qµ L we keep ncreasng qµ we wll get qµ L we cannot go beond L would move to g > ts s eactl te denton o te dual problem ma q µ µ note: [ L, µ ] mn[ g ] q µ mn + µ wt µ qµ ma go to - or some µ, wc means tat tere s no Lagrange multpler plane would be vertcal. ts s avoded b ntroducng te constrant { µ µ > } µ D q D q 6
Dualt we tereore ave a two step recpe to nd te optmal soluton 1.or an µ, solve [ L, µ ] mn[ µ g ] q µ mn +.ten solve ma q µ µ, µ Dq one o te reasons w ts s nterestng s tat te second problem turns out to be qute manageable 7
Dualt eorem: D q s a conve set and q s concave on D q Proo: or an, µ, µ and α [,1] L + 1 α µ + αµ + 1 α µ, αµ g + αµ g + 1 α µ g [ µ g ] 1 α [ µ g ] + + + α α L, µ + 1 α L, µ and takng mn on bot sdes mn L we ave, αµ + 1 α µ mn[ αl, µ + 1 α L, µ ] α mn L, µ + 1 α mn L, µ αµ + 1 α µ αq µ + 1 α q µ q 8
Dualt we ave αµ + 1 α µ αq µ + 1 α q µ q rom wc two conclusons ollow µ D q and µ D q qµ>-, qµ>- αµ+ 1-αµ D q Hence D q s sconve b denton o concavt mples tat q s concave over D q ts s a ver appealng result, snce conve optmzaton problems are among te easest to solve note tat te dual s alwas concave, rrespectve o te prmal problem te ollowng result onl proves wat we alread ave nerred rom te grapcal nterpretaton 9
Dualt eorem: weak dualt t s alwas true tat Proo: q or an µ, and wt g, q µ mn L z, µ + µ g z snce µ g. Hence q ma q µ µ and snce ts olds or an q mn, g 3
Dualt gap q we sa tat tere s no dualt gap. Oterwse tere s a dualt gap. te DG constrans te estence o Lagrange multplers eorem: tere s no dualt gap, te set o Lagrange multplers s te set o optmal dual solutons tere s a dualt gap, tere are no Lagrange multplers l Proo: b denton, µ µ s a Lagrange multpler and onl q µ q wc, rom te prevous teorem, olds and onl q,.e. tere s no dualt gap 31
Dualt gap note tat tere are stuatons n wc te dual problem as a soluton, but or wc tere s no Lagrange g multpler. or eample: ts s a vald dual problem owever te constrant µ g s not satsed and R, q g < g R r-1 n summar, dualt s nterestng onl wen tere s no dualt gap,q 3
33