Duality revisited Javier Peña Conve Optimization 10-725/36-725 1
Last time: barrier method Main idea: approimate the problem f() + I C () with the barrier problem f() + 1 t φ() tf() + φ() where t > 0 and φ is a barrier function for C. 2
Logarithmic barrier Common case: Logarithmic barrier C = { : h i () 0, i = 1,..., m}. φ() = m log( h i ()) i=1 Nice fact: when f, h i are smooth and KKT hold for both original and barrier problem, the solution (t) to barrier problem satisfies f( (t)) f m/t. 3
Strict feasibility Important detail: throughout the algorithm, line-search should be performed so that the iterates satisfy h i () < 0, i = 1,... m. To find an initial for the barrier problem solve,s s h i () s, i = 1,... m. Stop early: as soon as we find a feasible solution with s < 0. If the above imum is positive, then original problem is infeasible. 4
Outline Today Lagrangian duality revisited Optimality conditions Connection with barrier problems Fenchel duality 5
Lagrangian duality revisited Consider the primal problem f() h i () 0, i = 1,... m l j () = 0, j = 1,... r Lagrangian m r L(, u, v) = f() + u i h i () + v j l j () We can rewrite the primal problem as i=1 j=1 Dual problem ma u,v u 0 ma u,v u 0 L(, u, v) L(, u, v) 6
Weak and strong duality Theorem (weak duality) Let p and d denote the optimal values of the above primal and dual problems respectively. Then p d. Theorem (strong duality) Assume f, h 1,..., h p are conve with domain D and h p+1,..., h m, l 1,..., l r are affine. If there eists ˆ relint(d) such that h i (ˆ) < 0, i = 1,..., p; h i (ˆ) 0, i = p + 1,..., m and l j (ˆ) = 0, j = 1,..., r then p = d and the dual optimum is attained if finite. 7
Eample: linear programg Primal problem (in standard form) c T 0 Dual problem ma y,s b T y A T y + s = c s 0 8
Eample: conve quadratic programg Primal problem (in standard form) 1 2 T Q + c T 0 where Q symmetric and positive semidefinite. Dual problem ma u,y,s b T y 1 2 ut Qu A T y + s c = Qu s 0 9
10 Eample: barrier problem for linear programg Primal problem c T τ n log( i ) i=1 where τ > 0. Dual problem ma y,s n b T y + τ log(s i ) + n(τ τ log τ) i=1 A T y + s = c
Optimality conditions Consider the problem f() h() 0 Here h() 0 is shorthand for h i () 0, i = 1,... m. Assume f, h 1,..., h m are conve and differentiable. Assume also that strong duality holds. Then and (u, v ) are respectively primal and dual optimal solutions if and only if (, u, v ) solves the KKT conditions f() + A T v + h()u = 0 Uh() = 0 u, h() 0. Here U = Diag(u), h() = [ h 1 () h m () ] 11
12 Barrier problem where Central path equations φ() = f() + τφ() m log( h i ()). i=1 Optimality conditions for barrier problem (and its dual) f() + A T v + h()u = 0 Uh() = τ1 u, h() > 0.
13 Special case: linear programg Primal and dual problems c T 0 ma y,s b T y A T y + s = c s 0 Optimality conditions for both A T y + s = c XS1 = 0, s 0. Here X = Diag(), S = Diag(s).
14 Algorithms for linear programg Recall the optimality conditions for linear programg Two main classes of algorithms A T y + s = c XS1 = 0, s 0. Simple: maintain first three and aim for fourth one Interior-point methods: maintain fourth (and maybe first and second) and aim for third one.
15 Central path for linear programg Primal and dual barrier problems c T τ n log( i ) i=1 ma y,s n b T y + τ log(s i ) i=1 A T y + s = c Optimality conditions for both A T y + s = c XS1 = τ1, s > 0.
16 Fenchel duality Consider the primal problem f() + g(a) Rewrite it as Dual problem f() + g(z) A = z. ma v f (A T v) g ( v). This special type of duality is called Fenchel duality. Nice fact: if f, g are conve and closed then the dual of the dual is the primal.
17 Eample: conic programg Primal problem (in standard form) c T K where K is a closed conve cone. Dual problem ma y,s b T y A T y + s = c s K Strong duality holds if one of the problems is strictly feasible. If both primal and dual are strictly feasible, then strong duality holds and both primal and dual optima are attained.
18 Eample: semidefinite programg Primal Dual X ma y C X A i X = b i, i = 1,..., m X 0. b T y m y i A i + S = C i=1 S 0. Recall trace inner product in S n X S = trace(xs).
19 Strong duality does not always hold Eamples [ 2 12 ] 0 12 0. 12 22 [11 11 ] 1 0. 1 22 a 22 0 12 1 22 12 22 23 0, for a > 0. 1 22 23 33
20 Eample: barrier problem for semidefinite programg Primal X C X τ log(det(x)) A i X = b i, i = 1,..., m Dual ma y,s b T y + τ log(det(s)) + n(τ τ log τ) m y i A i + S = C i=1
21 Optimality conditions for semidefinite programg Primal and dual problems X C X A(X) = b X 0 ma y,s b T y A (y) + S = C C 0 Here A : S n R m linear map. Assume also that strong duality holds. Then X and (y, S ) are respectively primal and dual optimal solutions if and only if (X, y, S ) solves A (y) + S = C A(X) = b XS = 0 X, S 0.
22 Central path for semidefinite programg Primal barrier problem X C X τ log(det(x)) A(X) = b Dual barrier problem ma y,s b T y + τ log(det(s)) A (y) + S = C Optimality conditions for both A (y) + S = C A(X) = b XS = τi X, S 0.
23 References O. Guler (2010), Foundations of Optimization, Chapter 11 J. Renegar (2001), A mathematical view of interior-point methods in conve optimization, Chapters 2 and 3. S. Wright (1997), Primal-dual interior-point methods, Chapters 5 and 6.