Linear and Combinatorial Optimization The dual of an LP-problem. Connections between primal and dual. Duality theorems and complementary slack. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality I 28. 1. 14 1 / 20
The dual of an LP-problem Definition 3.1 Let max z = c T x (P) Ax b x 0 be an LP-problem in standard form (primal). The dual to the above problem is defined as min v = b T y (D) A T y c y 0 Theorem 3.1 The dual of its dual is again the primal problem. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality I 28. 1. 14 2 / 20
Other Primal-Dual pairs Theorem 3.2 The dual to a canonical LP-problem max z = c T x (P) Ax = b x 0 is min v = b T y (D) A T y c y unconstrained Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality I 28. 1. 14 3 / 20
Other Primal-Dual pairs Theorem 3.3 The dual to the LP-problem max z = c T x (P) Ax b x 0 is min v = b T y (D) A T y c y 0 Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality I 28. 1. 14 4 / 20
Primal-Dual Table Primal Dual max min Ax b y 0 Ax = b y unc. Ax b y 0 x 0 A T y c x unc. A T y = c x 0 A T y c Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality I 28. 1. 14 5 / 20
Interpretation of dual variables The dual variable measures how active the corresponding constraint is. When the constraint is not active (a T i x < b i ) the corresponding dual variable y i = 0. Interpretations: Reaction forces Economic interpretation Shadow prices Accounting prices Marginal value Replacement value. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality I 28. 1. 14 6 / 20
Weak Duality Theorem For both primal and dual, there are three possible outcomes: No feasible solution exists. Finite optimum. Feasible solutions, but objective function unbounded. Theorem 3.4 (Weak Duality) If x is a feasible solution to the primal problem and y is a feasible solution to the dual problem, then c T x b T y. Proof 1 Since Ax b and A T y c (and x 0, y 0) then c T x (A T y) T x = y T Ax y T b = b T y. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality I 28. 1. 14 7 / 20
Corollary to the Weak Duality Theorem Corollary 3.1 (a) If the primal problem has a feasible solution, but the objective function is unbounded, then the dual has no feasible solution. (b) If the dual problem has a feasible solution, but the objective function is unbounded, then the primal has no feasible solution. Proof 2 c T x b T y Observe that it may happen that both the primal and dual has no feasible solution! Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality I 28. 1. 14 8 / 20
Theorem 3.5 If x and y are feasible solutions to the primal and the dual, respectively, and if the objective function values are equal c T x = b T y, then both x and y are optimal solution to their problems. Proof 3 for all feasible ˆx and ŷ, implying c T ˆx b T y = c T x b T ŷ c T ˆx T c T x and b T y T b T ŷ i.e. x and y are optimal. feasible ˆx feasible ŷ Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality I 28. 1. 14 9 / 20
Complementary Slackness Definition 3.2 Let x denote a feasible solution to the primal problem and y a feasible solution to the dual problem. The primal solution x and the dual solution y fulfill the complementary slackness condition (CS) if y T }{{} 0 (Ax b) = 0 }{{}}{{} x T 0 0 Interpretation: Ax b active or y = 0. Lemma 3.1 If x, y fulfill CS then Proof. c T x = B T y. (A T y c) = 0 }{{} 0 CS implies that y T Ax = y T b = b T y and x T A T y = c T x implies c T x = b T y. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality I 28. 1. 14 10 / 20
Theorem 3.6 If x is optimal for (P) and y is optimal for (D), then CS holds, i.e. y T (Ax b) = 0 and x T (A T y c) = 0 Remark: Let x be feasible, but non-optimal, i.e. c T x c T x. Construct a dual y based on CS. b T y = c T x c T x = b T y y is not feasible! x feasible non-optimal for P y good, but not feasible for D y feasible non-optimal for D x good, but not feasible for P Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality I 28. 1. 14 11 / 20
The Strong Duality Theorem Theorem 3.7 (Strong duality) If x is optimal for (P) and y is optimal for (D), then c T x = b T y. The previous Theorem now follows directly: Theorem 3.8 (Theorem 3.6) If x is optimal for (P) and y is optimal for (D), then CS holds. Proof 4 Let x and y be optimal solutions to (P) and (D). The strong duality theorem gives that c T x = b T y, which implies y T }{{} 0 (Ax b) = }{{}}{{} x T 0 0 (A T y c) = 0. }{{} 0 Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality I 28. 1. 14 12 / 20
The Complete Duality Theorem Theorem 3.9 The complete duality theorem (i) If x is feasible and optimal for (P) then there exists a y that is feasible and optimal for (D) and c T x = b T y. (ii) If (P) is unbounded then (D) has no feasible solution. (iii) If (P) has no feasible solution then (D) is either unbounded or has no feasible solution. Proof: (i) See the construction is previous theorem. (ii) See previous corollary (iii) By duality Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality I 28. 1. 14 13 / 20
Working with the Primal or Dual Observe that the optimal solution x = [xb x N ] fulfills (with A = [B N]). { x B = B 1 b and x N = 0 y = B T c B z = c T B B 1 b The primal problem aims at solving Bx B = b. The dual problem aims at solving B T y = c B. Sometimes it is advantageous to work with (P), sometimes with (D). There also exists a dual simplex method and even a primal-dual method! Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality I 28. 1. 14 14 / 20
Duality in general Consider an optimization of the form: p - Optimal objective value Primal problem Assume nothing about convexity no equality constraints The Lagrange function is min f 0 (x) (1) f i (x) 0, i = 1,... m. (2) L(x, w) = f 0 (x) + m w i f i (x) w i - Lagrange multipliers or dual variables. The objective function is extended with weighted constraint functions. i=1 Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality I 28. 1. 14 15 / 20
The Lagrange dual function The Lagrange dual function is g(w) = inf x L(x, w) = inf (f 0(x) + x Can attain for certain w. NB: g is convex even if f i are not. m w i f i (x)) i=1 Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality I 28. 1. 14 16 / 20
Example: LP min c T x (3) a T i x b i 0, i = 1,... m. (4) m L(x, w) = c T x + w i (ai T x b i ) i=1 = b T w + (A T w + c) T x g(w) = { b T w if A T w + c = 0 otherwise Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality I 28. 1. 14 17 / 20
The dual function bounds the primal problem Same idea as for LP if w 0 and x is feasible, then g(w) f 0 (x) f 0 (x) f 0 (x) + w i f i (x) inf z (f 0(z) + w i f i (z)) = g(w) f 0 (x) g(w) - Duality gap w is called feasible if w 0 and g(w) >. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality I 28. 1. 14 18 / 20
The dual optimization problem Find the best lower bound for p. max g(w) (5) w 0. (6) called the (Lagrange) dual problem to the primal problem is always a convex problem even if primal is not! d - optimal value. d p (weak duality) p d - optimal duality gap. Strong duality For convex problems, usually one has p = d Duality is particularly useful for convex problems. Non convex problems have generally not strong duality. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality I 28. 1. 14 19 / 20
Repetition - Lecture 3 The dual to an LP-problem. Interpretation of dual variables. c T x b T w, with equality iff x,w are optimal solutions. The Duality theorem. Complementary slackness. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality I 28. 1. 14 20 / 20