14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.

Transcription

1 CS/ECE/ISyE 524 Introduction to Optimization Spring Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity analysis Laurent Lessard (

2 Upper bounds Optimization problem (not necessarily convex!): x D f 0 (x) subject to: f i (x) 0 for i = 1,..., m h j (x) = 0 for j = 1,..., r 14-2

3 Upper bounds Optimization problem (not necessarily convex!): x D f 0 (x) subject to: f i (x) 0 for i = 1,..., m h j (x) = 0 for j = 1,..., r ˆ D is the domain of all functions involved. ˆ Suppose the optimal value is p. 14-2

4 Upper bounds Optimization problem (not necessarily convex!): x D f 0 (x) subject to: f i (x) 0 for i = 1,..., m h j (x) = 0 for j = 1,..., r ˆ D is the domain of all functions involved. ˆ Suppose the optimal value is p. ˆ Upper bounds: if x D satisfies f i (x) 0 and h j (x) = 0 for all i and j, then: p f 0 (x). 14-2

5 Upper bounds Optimization problem (not necessarily convex!): x D f 0 (x) subject to: f i (x) 0 for i = 1,..., m h j (x) = 0 for j = 1,..., r ˆ D is the domain of all functions involved. ˆ Suppose the optimal value is p. ˆ Upper bounds: if x D satisfies f i (x) 0 and h j (x) = 0 for all i and j, then: p f 0 (x). ˆ Any feasible x yields an upper bound for p. 14-2

6 Lower bounds Optimization problem (not necessarily convex!): x D f 0 (x) subject to: f i (x) 0 for i = 1,..., m h j (x) = 0 for j = 1,..., r ˆ As with LPs, use the constraints to find lower bounds 14-3

7 Lower bounds Optimization problem (not necessarily convex!): x D f 0 (x) subject to: f i (x) 0 for i = 1,..., m h j (x) = 0 for j = 1,..., r ˆ As with LPs, use the constraints to find lower bounds ˆ For any λ i 0 and ν j R, if x D is feasible, then f 0 (x) f 0 (x) + m λ i f i (x) + i=1 r ν j h j (x) j=1 14-3

8 Lower bounds f 0 (x) f 0 (x) + m λ i f i (x) + i=1 r ν j h j (x) j=1 } {{ } Lagrangian L(x, λ, ν) This is a lower bound on f 0, but we want a lower bound on p. Minimize right side over x D and left side over feasible x. 14-4

9 Lower bounds f 0 (x) f 0 (x) + m λ i f i (x) + i=1 r ν j h j (x) j=1 } {{ } Lagrangian L(x, λ, ν) This is a lower bound on f 0, but we want a lower bound on p. Minimize right side over x D and left side over feasible x. p { inf x D } L(x, λ, ν) = g(λ, ν) This inequality holds whenever λ

10 Lower bounds L(x, λ, ν) := f 0 (x) + m λ i f i (x) + i=1 r ν j h j (x) j=1 Whenever λ 0, we have: g(λ, ν) := { inf x D } L(x, λ, ν) p Useful fact: g(λ, ν) is a concave function. This is true even if the original optimization problem is not convex! (because g is a pointwise minimum of affine functions) 14-5

11 General duality Primal problem (P) x D f 0 (x) subject to: f i (x) 0 i h j (x) = 0 j Dual problem (D) maximize λ,ν g(λ, ν) subject to: λ

12 General duality Primal problem (P) x D f 0 (x) subject to: f i (x) 0 i h j (x) = 0 j Dual problem (D) maximize λ,ν g(λ, ν) subject to: λ 0 If x and λ are feasible points of (P) and (D) respectively: g(λ, ν) d p f 0 (x) 14-6

13 General duality Primal problem (P) x D f 0 (x) subject to: f i (x) 0 i h j (x) = 0 j Dual problem (D) maximize λ,ν g(λ, ν) subject to: λ 0 If x and λ are feasible points of (P) and (D) respectively: g(λ, ν) d p f 0 (x) This is called the Lagrange dual. Bad news: strong duality (p = d ) does not always hold! 14-6

14 Example (Srikant) x x subject to: (x 2)(x 4) x -5 x (x - 2) (x - 4) 14-7

15 Example (Srikant) x x subject to: (x 2)(x 4) x -5 x (x - 2) (x - 4) ˆ optimum occurs at x = 2, has value p =

16 Example (Srikant) Lagrangian: L(x, λ) = x λ(x 2)(x 4) x -5 ˆ Plot for different values of λ 0 ˆ Lower bounds for x on the feasible set 2 x

17 Example (Srikant) Lagrangian: L(x, λ) = x λ(x 2)(x 4) ˆ Minimize the Lagrangian: g(λ) = inf x = inf x L(x, λ) (λ + 1)x 2 6λx + (8λ + 1) If λ 1, it is unbounded. If λ > 1, the minimum occurs when 2(λ + 1)x 6λ = 0, so ˆx = 3λ g(λ) = { 9λ 2 /(1 + λ) λ λ+1. λ > 1 λ

18 Example (Srikant) maximize λ 9λ 2 /(1 + λ) λ subject to: λ 0 g(λ) λ

19 Example (Srikant) maximize λ 9λ 2 /(1 + λ) λ subject to: λ 0 g(λ) λ ˆ optimum occurs at λ = 2, has value d = 5 ˆ same optimal value as primal problem! (strong duality) 14-10

20 Constraint qualifications ˆ weak duality (d p ) always holds. Even when the optimization problem is not convex. ˆ strong duality (d = p ) often holds for convex problems (but not always)

21 Constraint qualifications ˆ weak duality (d p ) always holds. Even when the optimization problem is not convex. ˆ strong duality (d = p ) often holds for convex problems (but not always). A constraint qualification is a condition that guarantees strong duality

22 Constraint qualifications ˆ weak duality (d p ) always holds. Even when the optimization problem is not convex. ˆ strong duality (d = p ) often holds for convex problems (but not always). A constraint qualification is a condition that guarantees strong duality. An example we ve already seen: ˆ If the optimization problem is an LP, strong duality holds 14-11

23 Slater s constraint qualification x D f 0 (x) subject to: f i (x) 0 for i = 1,..., m h j (x) = 0 for j = 1,..., r Slater s constraint qualification: If the optimization problem is convex and strictly feasible, then strong duality holds. ˆ convexity requires: D and f i are convex and h j are affine. ˆ strict feasibility means there exists some x in the interior of D such that f i ( x) < 0 for i = 1,..., m

24 Slater s constraint qualification If the optimization problem is convex and strictly feasible, then strong duality holds. ˆ Good news: Slater s constraint qualification is rather weak. i.e. it is usually satisfied by convex problems. ˆ Can be relaxed so that strict feasibility is not required for the linear constraints

25 Counterexample (Boyd) x R, y>0 e x subject to: x 2 /y 0 ˆ The function x 2 /y is convex for y > 0 (see plot) ˆ The objective e x is convex ˆ Feasible set: {(0, y) y > 0} ˆ Solution is trivial (p = 1) 14-14

26 Counterexample (Boyd) x R, y>0 e x subject to: x 2 /y 0 ˆ Lagrangian: L(x, y, λ) = e x + λx 2 /y 14-15

27 Counterexample (Boyd) x R, y>0 e x subject to: x 2 /y 0 ˆ Lagrangian: L(x, y, λ) = e x + λx 2 /y ˆ Dual function: g(λ) = inf x,y>0 (e x + λx 2 /y) =

28 Counterexample (Boyd) x R, y>0 e x subject to: x 2 /y 0 ˆ Lagrangian: L(x, y, λ) = e x + λx 2 /y ˆ Dual function: g(λ) = inf x,y>0 (e x + λx 2 /y) = 0. ˆ The dual problem is: maximize λ 0 0 So we have d = 0 < 1 = p

29 Counterexample (Boyd) x R, y>0 e x subject to: x 2 /y 0 ˆ Lagrangian: L(x, y, λ) = e x + λx 2 /y ˆ Dual function: g(λ) = inf x,y>0 (e x + λx 2 /y) = 0. ˆ The dual problem is: maximize λ 0 0 So we have d = 0 < 1 = p. ˆ Slater s constraint qualification is not satisfied! 14-15

30 About Slater s constraint qualification Slater s condition is only sufficient. (Slater) = (strong duality) 14-16

31 About Slater s constraint qualification Slater s condition is only sufficient. (Slater) = (strong duality) ˆ There exist problems where Slater s condition fails, yet strong duality holds. ˆ There exist nonconvex problems with strong duality

32 Complementary slackness Assume strong duality holds. If x is primal optimal and (λ, ν ) is dual optimal, then we have: g(λ, ν ) = d = p = f 0 (x ) 14-17

33 Complementary slackness Assume strong duality holds. If x is primal optimal and (λ, ν ) is dual optimal, then we have: f 0 (x ) = g(λ, ν ) = inf x D g(λ, ν ) = d = p = f 0 (x ) ( f 0 (x ) + f 0 (x ) f 0 (x) + m λ i f i (x) + i=1 m λ i f i (x ) + i=1 ) r νj h j (x) j=1 r νj h j (x ) The last inequality holds because x is primal feasible. We conclude that the inequalities must all be equalities. j=

34 Complementary slackness ˆ We concluded that: f 0 (x ) = f 0 (x ) + m λ i f i (x ) + i=1 r νj h j (x ) j=1 But f i (x ) 0 and h j (x ) = 0. Therefore: λ i f i (x ) = 0 for i = 1,..., m 14-18

35 Complementary slackness ˆ We concluded that: f 0 (x ) = f 0 (x ) + m λ i f i (x ) + i=1 r νj h j (x ) j=1 But f i (x ) 0 and h j (x ) = 0. Therefore: λ i f i (x ) = 0 for i = 1,..., m ˆ This property is called complementary slackness. We ve seen it before for linear programs

36 Complementary slackness ˆ We concluded that: f 0 (x ) = f 0 (x ) + m λ i f i (x ) + i=1 r νj h j (x ) j=1 But f i (x ) 0 and h j (x ) = 0. Therefore: λ i f i (x ) = 0 for i = 1,..., m ˆ This property is called complementary slackness. We ve seen it before for linear programs. λ i > 0 = f i (x ) = 0 and f i (x ) < 0 = λ i =

37 Dual of an LP x 0 subject to: c T x Ax b 14-19

38 Dual of an LP x 0 subject to: c T x Ax b ˆ Lagrangian: L(x, λ) = c T x + λ T (b Ax) 14-19

39 Dual of an LP x 0 subject to: c T x Ax b ˆ Lagrangian: L(x, λ) = c T x + λ T (b Ax) ˆ Dual function: g(λ) = min x 0 (c AT λ) T x + λ T b 14-19

40 Dual of an LP x 0 subject to: c T x Ax b ˆ Lagrangian: L(x, λ) = c T x + λ T (b Ax) ˆ Dual function: g(λ) = min x 0 (c AT λ) T x + λ T b g(λ) = { λ T b if A T λ c otherwise 14-19

41 Dual of an LP x 0 subject to: c T x Ax b ˆ Dual is: maximize λ 0 subject to: λ T b A T λ c ˆ This is the same result that we found when we were studying duality for linear programs

42 Dual of an LP What if we treat x 0 as a constraint instead? (D = R n ). x subject to: c T x Ax b x

43 Dual of an LP What if we treat x 0 as a constraint instead? (D = R n ). x subject to: c T x Ax b x 0 ˆ Lagrangian: L(x, λ, µ) = c T x + λ T (b Ax) µ T x 14-21

44 Dual of an LP What if we treat x 0 as a constraint instead? (D = R n ). x subject to: c T x Ax b x 0 ˆ Lagrangian: L(x, λ, µ) = c T x + λ T (b Ax) µ T x ˆ Dual function: g(λ, µ) = min (c A T λ µ) T x + λ T b x 14-21

45 Dual of an LP What if we treat x 0 as a constraint instead? (D = R n ). x subject to: c T x Ax b x 0 ˆ Lagrangian: L(x, λ, µ) = c T x + λ T (b Ax) µ T x ˆ Dual function: g(λ, µ) = min (c A T λ µ) T x + λ T b x { λ T b if A T λ + µ = c g(λ) = otherwise 14-21

46 Dual of an LP What if we treat x 0 as a constraint instead? (D = R n ). x subject to: c T x Ax b x 0 ˆ Dual is: maximize λ 0, µ 0 subject to: λ T b A T λ + µ = c ˆ Solution is the same, µ acts as the slack variable

47 Dual of a convex QP Suppose Q 0. Let s find the dual of the QP: x subject to: 1 x T Qx 2 Ax b 14-23

48 Dual of a convex QP Suppose Q 0. Let s find the dual of the QP: x subject to: 1 x T Qx 2 Ax b ˆ Lagrangian: L(x, λ) = 1 2 x T Qx + λ T (b Ax) 14-23

49 Dual of a convex QP Suppose Q 0. Let s find the dual of the QP: x subject to: 1 x T Qx 2 Ax b ˆ Lagrangian: L(x, λ) = 1x T Qx + λ T (b Ax) 2 ( ˆ Dual function: g(λ) = min 1 x T Qx + λ T (b Ax) ) x 2 Minimum occurs at: ˆx = Q 1 A T λ 14-23

50 Dual of a convex QP Suppose Q 0. Let s find the dual of the QP: x subject to: 1 x T Qx 2 Ax b ˆ Lagrangian: L(x, λ) = 1x T Qx + λ T (b Ax) 2 ( ˆ Dual function: g(λ) = min 1 x T Qx + λ T (b Ax) ) x 2 Minimum occurs at: ˆx = Q 1 A T λ g(λ) = 1 2 λt AQ 1 A T λ + λ T b 14-23

51 Dual of a convex QP Suppose Q 0. Let s find the dual of the QP: x subject to: 1 x T Qx 2 Ax b ˆ Dual is also a QP: maximize λ 1 2 λt AQ 1 A T λ + λ T b subject to: λ 0 ˆ It s still easy to solve (maximizing a concave function) 14-24

52 Sensitivity analysis min x D f 0 (x) s.t. f i (x) u i i h j (x) = v j j max λ,ν g(λ, ν) λ T u ν T v s.t. λ 0 ˆ As with LPs, dual variables quantify the sensitivity of the optimal cost to changes in each of the constraints

53 Sensitivity analysis min x D f 0 (x) s.t. f i (x) u i i h j (x) = v j j max λ,ν g(λ, ν) λ T u ν T v s.t. λ 0 ˆ As with LPs, dual variables quantify the sensitivity of the optimal cost to changes in each of the constraints. ˆ A change in u i causes a bigger change in p if λ i ˆ A change in v j causes a bigger change in p if ν j is larger. is larger

54 Sensitivity analysis min x D f 0 (x) s.t. f i (x) u i i h j (x) = v j j max λ,ν g(λ, ν) λ T u ν T v s.t. λ 0 ˆ As with LPs, dual variables quantify the sensitivity of the optimal cost to changes in each of the constraints. ˆ A change in u i causes a bigger change in p if λ i ˆ A change in v j causes a bigger change in p if ν j is larger. is larger. ˆ If p (u, v) is differentiable, then: λ i = p (0, 0) u i and ν j = p (0, 0) v j 14-25

55 Next class... ˆ Review! 14-26