LECTURE 24 LECTURE OUTLINE Gradet proxmal mmzato method Noquadratc proxmal algorthms Etropy mmzato algorthm Expoetal augmeted Lagraga mehod Etropc descet algorthm ************************************** Refereces: Bec, A., ad Teboulle, M., 200. Gradet- Based Algorthms wth Applcatos to Sgal Recovery Problems, Covex Optmzato Sgal Processg ad Commucatos (Y. Eldar ad D. Palomar, eds.), Cambrdge Uversty Press, pp. 42-88. Bec, A., ad Teboulle, M., 2003. Mrror Descet ad Nolear Projected Subgradet Methods for Covex Optmzato, Operatos Research Letters, Vol. 3, pp. 67-75. Bertseas, D. P., 999. Nolear Programmg, Athea Scetfc, Belmot, MA. All fgures are courtesy of Athea Scetfc, ad are used wth permsso.
PROXIMAL AND GRADIENT PROJECTION Proxmal algorthm to mmze covex f over closed covex X { x + arg m f(x)+ x x 2 x X 2c f(x ) γ f(x) γ 2c x x 2 x x + x x Let f be dfferetable ad assume f(x) f(y) L x y, x, y X Defe the lear approxmato fucto at x!(y; x) =f(x)+ f(x) (y x), y Coecto of proxmal wth gradet projecto { ( y = arg m!(z; x)+ z X 2α z x 2 = P X x α f(x) 2
GRADIENT-PROXIMAL METHOD I Mmze f(x)+g(x) over x X, where X: closed covex, f, g: covex, f s dfferetable. Gradet-proxmal method: { x + arg m!(x; x )+g(x)+ x X 2α x x 2 Recall ey equalty: For all x, y X L f(y)!(y; x)+ y x 2 2 Cost reducto for α /L: L f(x + )+g(x + )!(x + ; x )+ x 2 + x 2 + g(x + )!(x + ; x )+g(x + )+ x + x 2 2α!(x ; x )+g(x ) = f(x )+g(x ) Ths s a ey sght for the covergece aalyss. 3
GRADIENT-PROXIMAL METHOD II Equvalet defto of gradet-proxmal: x + arg m x X z = x α f(x ) { g(x)+ 2α x z 2 Smplfes the mplemetato of proxmal, by usg gradet terato to deal wth the case of a coveet compoet f Ths s smlar to cremetal subgradet-proxmal method, but the gradet-proxmal method does ot exted to the case where the cost cossts of the sum of multple compoets. Allows a costat stepsze (uder the restrcto α /L). Ths does ot exted to cremetal methods. Le all gradet ad subgradet methods, covergece ca be slow. There are specal cases where the method ca be frutfully appled (see the referece by Bec ad Teboulle). 4
GENERALIZED PROXIMAL ALGORITHM Itroduce a geeral regularzato term D : x + arg m x X Example: Bregma f(x)+d (x, x ) dstace fucto D (x, y) = ( (x) ( y) (y) (x c y), where : (, ] s a covex fucto, dfferetable wth a ope set cotag dom(f), ad c s a postve pealty parameter. All the deas for applcatos ad coectos of the quadratc form of the proxmal algorthm exted to the oquadratc case (although the aalyss may ot be trval). I partcular we have: A dual proxmal algorthm (based o Fechel dualty) Equvalece wth (oquadratc) augmeted Lagragea method Combatos wth polyhedral approxmatos (budle-type methods) Icremetal subgradet-proxmal methods Nolear gradet projecto algorthms 5
ENTROPY MINIMIZATION ALGORITHM A specal case volvg etropy regularzato: x x + arg m f(x)+ x l x X c x = where x 0 ad all subsequet x have postve compoets We use Fechel dualty to obta a dual form of ths mmzato Note: The logarthmc fucto p(x) = x(l x ) ad the expoetal fucto are a cojugate par. The dual problem s y + arg m f x>0, 0 f x =0, f x<0, p (y) =e y f (y)+ x e c y y c = 6
EXPONENTIAL AUGMENTED LAGRANGIAN The dual proxmal terato s x = x e c y + +, =,..., where y + s obtaed from the dual proxmal: y + arg m y f (y)+ c = x e c y A specal case for the covex problem mmze f(x) subject to g (x) 0,...,g r (x) 0, x X s the expoetal augmeted Lagragea method Cossts of ucostraed mmzatos r x arg m f(x)+ µ j e c g j (x), x X c j= followed by the multpler teratos j µ j j c g j (x ) + = µ e, j =,...,r 7
NONLINEAR PROJECTION ALGORITHM Subgradet projecto wth geeral regularzato term D : x + arg m f(x )+ f(x ) (x x )+D (x, x ) x X where f(x ) s a subgradet of f at x. called mrror descet method. Also Learzato of f smplfes the mmzato The use of oquadratc learzato s useful problems wth specal structure Etropc descet method: Mmze f(x) over the ut smplex X = x 0 x =. Method: x + arg m = x g + x X α = where g are the compoets of f(x ). l x x Ths mmzato ca be doe closed form: x + = x e α g, j =,..., x j e α g j= 8
MIT OpeCourseWare http://ocw.mt.edu 6.253 Covex Aalyss ad Optmzato Sprg 202 For formato about ctg these materals or our Terms of Use, vst: http://ocw.mt.edu/terms.