5. Duality. Lagrangian

Transcription

1 5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized inequalities 5 1 Lagrangian standard form problem (not necessarily convex) f 0 (x) f i (x) 0, i = 1,..., m h i (x) = 0, i = 1,..., p variable x R n, domain D, optimal value p Lagrangian: L : R n R m R p R, with dom L = D R m R p, L(x, λ, ν) = f 0 (x) + m λ i f i (x) + p ν i h i (x) weighted sum of objective and constraint functions λ i is Lagrange multiplier associated with f i (x) 0 ν i is Lagrange multiplier associated with h i (x) = 0 Duality 5 2

2 Lagrange Lagrange : g : R m R p R, g(λ, ν) = inf x D = inf x D L(x, λ, ν) ( f 0 (x) + m λ i f i (x) + ) p ν i h i (x) g is concave, can be for some λ, ν lower bound property: if λ 0, then g(λ, ν) p proof: if x is feasible and λ 0, then f 0 ( x) L( x, λ, ν) inf L(x, λ, ν) = g(λ, ν) x D minimizing over all feasible x gives p g(λ, ν) Duality 5 3 Least-norm solution of linear equations x T x Ax = b Lagrangian is L(x, ν) = x T x + ν T (Ax b) to L over x, set gradient equal to zero: x L(x, ν) = 2x + A T ν = 0 = x = (1/2)A T ν plug in in L to obtain g: g(ν) = L(( 1/2)A T ν, ν) = 1 4 νt AA T ν b T ν a concave function of ν lower bound property: p (1/4)ν T AA T ν b T ν for all ν Duality 5 4

3 Standard form LP c T x Ax = b, x 0 Lagrangian is L(x, λ, ν) = c T x + ν T (Ax b) λ T x = b T ν + (c + A T ν λ) T x L is linear in x, hence b g(λ, ν) = inf L(x, λ, ν) = T ν A T ν λ + c = 0 x otherwise g is linear on affine domain (λ, ν) A T ν λ + c = 0}, hence concave lower bound property: p b T ν if A T ν + c 0 Duality 5 5 Equality constrained norm minimization x Ax = b g(ν) = inf x ( x νt Ax + b T ν) = where v = sup u 1 u T v is dual norm of b T ν A T ν 1 otherwise proof: follows from inf x ( x y T x) = 0 if y 1, otherwise if y 1, then x y T x 0 for all x, with equality if x = 0 if y > 1, choose x = tu where u 1, u T y = y > 1: x y T x = t( u y ) as t lower bound property: p b T ν if A T ν 1 Duality 5 6

4 Two-way partitioning x T W x x 2 i = 1, i = 1,..., n a nonconvex problem; feasible set contains 2 n discrete points interpretation: partition 1,..., n} in two sets; W ij is cost of assigning i, j to the same set; W ij is cost of assigning to different sets g(ν) = inf x (xt W x + i ν i (x 2 i 1)) = inf x xt (W + diag(ν))x 1 T ν 1 = T ν W + diag(ν) 0 otherwise lower bound property: p 1 T ν if W + diag(ν) 0 example: ν = λ min (W )1 gives bound p nλ min (W ) Duality 5 7 Lagrange dual and conjugate function f 0 (x) Ax b, Cx = d g(λ, ν) = inf x dom f 0 ( f0 (x) + (A T λ + C T ν) T x b T λ d T ν ) = f 0 ( A T λ C T ν) b T λ d T ν recall definition of conjugate f (y) = sup x dom f (y T x f(x)) simplifies derivation of dual if conjugate of f 0 is kown example: entropy maximization f 0 (x) = n x i log x i, f0 (y) = n e y i 1 Duality 5 8

5 The dual problem Lagrange dual problem maximize g(λ, ν) λ 0 finds best lower bound on p, obtained from Lagrange a convex optimization problem; optimal value denoted d λ, ν are dual feasible if λ 0, (λ, ν) dom g often simplified by making implicit constraint (λ, ν) dom g explicit example: standard form LP and its dual (page 5 5) c T x Ax = b x 0 maximize b T ν A T ν + c 0 Duality 5 9 weak duality: d p Weak and strong duality always holds (for convex and nonconvex problems) can be used to find nontrivial lower bounds for difficult problems for example, solving the SDP maximize 1 T ν W + diag(ν) 0 gives a lower bound for the two-way partitioning problem on page 5 7 strong duality: d = p does not hold in general (usually) holds for convex problems conditions that guarantee strong duality in convex problems are called constraint qualifications Duality 5 10

6 Slater s constraint qualification strong duality holds for a convex problem if it is strictly feasible, i.e., f 0 (x) f i (x) 0, i = 1,..., m Ax = b x int D : f i (x) < 0, i = 1,..., m, Ax = b also guarantees that the dual optimum is attained (if p > ) can be sharpened: e.g., can replace int D with relint D (interior relative to affine hull); linear inequalities do not need to hold with strict inequality,... there exist many other types of constraint qualifications Duality 5 11 Inequality form LP primal problem c T x Ax b g(λ) = inf x ( (c + A T λ) T x b T λ ) b = T λ A T λ + c = 0 otherwise dual problem maximize b T λ A T λ + c = 0, λ 0 from Slater s condition: p = d if A x b for some x in fact, p = d except when primal and dual are infeasible Duality 5 12

7 primal problem (assume P S n ++) Quadratic program x T P x Ax b g(λ) = inf x ( x T P x + λ T (Ax b) ) = 1 4 λt AP 1 A T λ b T λ dual problem maximize (1/4)λ T AP 1 A T λ b T λ λ 0 from Slater s condition: p = d if A x b for some x in fact, p = d always Duality 5 13 A nonconvex problem with strong duality nonconvex if A 0 x T Ax + 2b T x x T x 1 : g(λ) = inf x (x T (A + λi)x + 2b T x λ) unbounded below if A + λi 0 or if A + λi 0 and b R(A + λi) d by x = (A + λi) b otherwise: g(λ) = b T (A + λi) b λ dual problem and equivalent SDP: maximize b T (A + λi) b λ A + λi 0 b R(A + λi) maximize t [ λ A + λi b b T t ] 0 strong duality although primal problem is not convex (not easy to show) Duality 5 14

8 Geometric interpretation for simplicity, consider problem with one constraint f 1 (x) 0 interpretation of : g(λ) = inf (t + λu), where G = (f 1(x), f 0 (x)) x D} (u,t) G PSfrag replacements PSfrag replacements f 1 (x) λu + t = g(λ) t G p g(λ) u t p d G u λu + t = g(λ) is (non-vertical) supporting hyperplane to G hyperplane intersects t-axis at t = g(λ) Duality 5 15 epigraph variation: same interpretation if G is replaced with A = (u, t) f 1 (x) u, f 0 (x) t for some x D} t PSfrag replacements A f 1 (x) λu + t = g(λ) p g(λ) u strong duality holds if there is a non-vertical supporting hyperplane to A at (0, p ) for convex problem, A is convex, hence has supp. hyperplane at (0, p ) Slater s condition: if there exist (ũ, t) A with ũ < 0, then supporting hyperplanes at (0, p ) must be non-vertical Duality 5 16

9 Complementary slackness assume strong duality holds, x is primal optimal, (λ, ν ) is dual optimal f 0 (x ) = g(λ, ν ) = inf x ( f 0 (x ) + f 0 (x ) f 0 (x) + m λ i f i (x) + m λ i f i (x ) + ) p νi h i (x) p νi h i (x ) hence, the two inequalities hold with equality x s L(x, λ, ν ) λ i f i(x ) = 0 for i = 1,..., m (known as complementary slackness): λ i > 0 = f i (x ) = 0, f i (x ) < 0 = λ i = 0 Duality 5 17 Karush-Kuhn-Tucker (KKT) conditions the following four conditions are called KKT conditions (for a problem with differentiable f i, h i ): 1. primal constraints: f i (x) 0, i = 1,..., m, h i (x) = 0, i = 1,..., p 2. dual constraints: λ 0 3. complementary slackness: λ i f i (x) = 0, i = 1,..., m 4. gradient of Lagrangian with respect to x vanishes: f 0 (x) + m λ i f i (x) + p ν i h i (x) = 0 from page 5 17: if strong duality holds and x, λ, ν are optimal, then they must satisfy the KKT conditions Duality 5 18

10 KKT conditions for convex problem if x, λ, ν satisfy KKT for a convex problem, then they are optimal: from complementary slackness: f 0 ( x) = L( x, λ, ν) from 4th condition (and convexity): g( λ, ν) = L( x, λ, ν) hence, f 0 ( x) = g( λ, ν) if Slater s condition is satisfied: x is optimal if and only if there exist λ, ν that satisfy KKT conditions recall that Slater implies strong duality, and dual optimum is attained generalizes optimality condition f 0 (x) = 0 for unconstrained problem Duality 5 19 example: water-filling (assume α i > 0) n log(x i + α i ) x 0, 1 T x = 1 x is optimal iff x 0, 1 T x = 1, and there exist λ R n, ν R such that λ 0, λ i x i = 0, 1 x i + α i + λ i = ν if ν < 1/α i : λ i = 0 and x i = 1/ν α i if ν 1/α i : λ i = ν 1/α i and x i = 0 determine ν from 1 T x = n max0, 1/ν α i} = 1 interpretation n patches; level of patch i is at height α i flood area with unit amount of water resulting level is 1/ν PSfrag replacements 1/ν x i Duality 5 20 i α i

11 Perturbation and sensitivity analysis (unperturbed) optimization problem and its dual f 0 (x) f i (x) 0, i = 1,..., m h i (x) = 0, i = 1,..., p maximize g(λ, ν) λ 0 perturbed problem and its dual min. f 0 (x) s.t. f i (x) u i, i = 1,..., m h i (x) = v i, i = 1,..., p max. g(λ, ν) u T λ v T ν s.t. λ 0 x is primal variable; u, v are parameters p (u, v) is optimal value as a function of u, v we are interested in information about p (u, v) that we can obtain from the solution of the unperturbed problem and its dual Duality 5 21 global sensitivity result assume strong duality holds for unperturbed problem, and that λ, ν are dual optimal for unperturbed problem apply weak duality to perturbed problem: p (u, v) g(λ, ν ) u T λ v T ν = p (0, 0) u T λ v T ν sensitivity interpretation if λ i large: p increases greatly if we tighten constraint i (u i < 0) if λ i small: p does not decrease much if we loosen constraint i (u i > 0) if ν i large and positive: p increases greatly if we take v i < 0; if ν i large and negative: p increases greatly if we take v i > 0 if ν i small and positive: p does not decrease much if we take v i > 0; if ν i small and negative: p does not decrease much if we take v i < 0 Duality 5 22

12 local sensitivity: if (in addition) p (u, v) is differentiable at (0, 0), then λ i = p (0, 0) u i, ν i = p (0, 0) v i proof (for λ i ): from global sensitivity result, hence, equality p (0, 0) u i p (0, 0) u i = lim t 0 p (te i, 0) p (0, 0) t = lim t 0 p (te i, 0) p (0, 0) t λ i λ i p PSfrag replacements (u) for a problem with one (inequality) constraint: u = 0 u p (u) p (0) λ u Duality 5 23 Duality and problem reformulations equivalent formulations of a problem can lead to very different duals reformulating the primal problem can be useful when the dual is difficult to derive, or uninteresting common reformulations introduce new variables and equality constraints make explicit constraints implicit or vice-versa transform objective or constraint functions e.g., replace f 0 (x) by φ(f 0 (x)) with φ convex, increasing Duality 5 24

13 Introducing new variables and equality constraints f 0 (Ax + b) is constant: g = inf x L(x) = inf x f 0 (Ax + b) = p we have strong duality, but dual is quite useless reformulated problem and its dual f 0 (y) Ax + b y = 0 maximize b T ν f0 (ν) A T ν = 0 follows from g(ν) = inf (f 0(y) ν T y + ν T Ax + b T ν) x,y f = 0 (ν) + b T ν A T ν = 0 otherwise Duality 5 25 norm approximation problem: Ax b y y = Ax b can look up conjugate of, or derive dual directly g(ν) = inf x,y νt y ν T Ax + b T ν) b = ν + inf y ( y + ν T y) A T ν = 0 otherwise b = ν A T ν = 0, ν 1 otherwise (see page 5 4) dual of norm approximation problem maximize b T ν A T ν = 0, ν 1 Duality 5 26

14 Implicit constraints LP with box constraints: primal and dual problem c T x Ax = b 1 x 1 maximize b T ν 1 T λ 1 1 T λ 2 c + A T ν + λ 1 λ 2 = 0 λ 1 0, λ 2 0 reformulation with box constraints made implicit c f 0 (x) = x 1 x 1 otherwise Ax = b g(ν) = inf 1 x 1 (ct x + ν T (Ax b)) = b T ν A T ν + c 1 dual problem: maximize b T ν A T ν + c 1 Duality 5 27 Problems with generalized inequalities f 0 (x) f i (x) Ki 0, i = 1,..., m h i (x) = 0, i = 1,..., p Ki is generalized inequality on R k i definitions are parallel to scalar case: Lagrange multiplier for f i (x) Ki 0 is vector λ i R k i Lagrangian L : R n R k 1 R k m R p R, is defined as L(x, λ 1,, λ m, ν) = f 0 (x) + m λ T i f i (x) + p ν i h i (x) g : R k 1 R k m R p R, is defined as g(λ 1,..., λ m, ν) = inf x D L(x, λ 1,, λ m, ν) Duality 5 28

15 lower bound property: if λ i K i 0, then g(λ 1,..., λ m, ν) p proof: if x is feasible and λ K i f 0 ( x) f 0 ( x) + 0, then m λ T i f i ( x) + inf L(x, λ 1,..., λ m, ν) x D = g(λ 1,..., λ m, ν) p ν i h i ( x) minimizing over all feasible x gives p g(λ 1,..., λ m, ν) dual problem weak duality: p d always maximize g(λ 1,..., λ m, ν) λ i K i 0, i = 1,..., m strong duality: p = d for convex problem with constraint qualification (for example, Slater s: primal problem is strictly feasible) Duality 5 29 primal SDP (F i, G S k ) Semidefinite program c T x x 1 F x n F n G Lagrange multiplier is matrix Z S k Lagrangian L(x, Z) = c T x + tr (Z(x 1 F x n F n G)) tr(gz) tr(fi Z) + c g(z) = inf L(x, Z) = i = 0, i = 1,..., n x otherwise dual SDP maximize tr(gz) Z 0, tr(f i Z) + c i = 0, i = 1,..., n p = d if primal SDP is strictly feasible ( x with x 1 F x n F n G) Duality 5 30