BBM402-Lecture 20: LP Duality - PDF Free Download

BBM402-Lecture 20: LP Duality Lecturer: Lale Özkahya Resources for the presentation: https://courses.engr.illinois.edu/cs473/fa2016/lectures.html

An easy LP? which is compact form for max cx subject to Ax = b max c 1 x 1 + c 2 x 2 +... + c n x n a i 1 x 1 + a i 2 x 2 +... + a in x n = b i 1 i m Question: Is this a geneal LP problem or is it some how easy? Chandra & Ruta (UIUC) CS473 3 Fall 2016 3 / 39

An easy LP? Basically reduces to linear system solving. Three cases for Ax = b. The system Ax = b is infeasible, that is, no solution The system Ax = b has a unique solution x when rank([a b]) = n (full rank). Optimum solution value is cx The system Ax = b has infinite solutions when rank([a b]) < n. There all vectors of the form x + y are feasible where y is null-space(a) = {y Ay = 0}. Let d be dimension of null-space(a) and let e 1, e 2,..., e d be an orthonormal basis. Then cx = cx + cy = cx + c(λ 1 e 1 + λ 2 e 2 +... + λ d e d. If ce i 0 for any i then optimum solution value is unbounded. Otherwise cx. Chandra & Ruta (UIUC) CS473 4 Fall 2016 4 / 39 max cx subject to Ax = b

LP Canonical Forms Two basic canonical forms: max cx, Ax = b, x 0 max cx, Ax b, x 0 What makes LP non-trivial and different from linear system solving is the additional non-negativity constraint on variables. Chandra & Ruta (UIUC) CS473 5 Fall 2016 5 / 39

Part I Derivation and Definition of Dual LP Chandra & Ruta (UIUC) CS473 6 Fall 2016 6 / 39

Feasible Solutions and Lower Bounds Consider the program maximize 4x 1 + 2x 2 subject to x 1 + 3x 2 5 2x 1 4x 2 10 x 1 + x 2 7 x 1 5 Chandra & Ruta (UIUC) CS473 7 Fall 2016 7 / 39

Feasible Solutions and Lower Bounds Consider the program maximize 4x 1 + 2x 2 subject to x 1 + 3x 2 5 2x 1 4x 2 10 x 1 + x 2 7 x 1 5 1 (0, 1) satisfies all the constraints and gives value 2 for the objective function. 2 Thus, optimal value σ is at least 4. Chandra & Ruta (UIUC) CS473 7 Fall 2016 7 / 39

Obtaining Upper Bounds 1 Let us multiply the first constraint by 2 and the and add it to second constraint 2( x 1 + 3x 2 ) 2(5) +1( 2x 1 4x 2 ) 1(10) 4x 1 + 2x 2 20 2 Thus, 20 is an upper bound on the optimum value! Chandra & Ruta (UIUC) CS473 8 Fall 2016 8 / 39

Generalizing... 1 Multiply first equation by y 1, second by y 2, third by y 3 and fourth by y 4 (all of y 1, y 2, y 3, y 4 being positive) and add y 1 ( x 1 + 3x 2 ) y 1 (5) +y 2 ( 2x 1 4x 2 ) y 2 (10) +y 3 ( x 1 + x 2 ) y 3 (7) +y 4 ( x 1 ) y 4 (5) (y 1 + 2y 2 + y 3 + y 4 )x 1 + (3y 1 4y 2 + y 3 )x 2... 2 5y 1 + 10y 2 + 7y 3 + 5y 4 is an upper bound, provided coefficients of x i are same as in the objective function, i.e., y 1 + 2y 2 + y 3 + y 4 = 4 3y 1 4y 2 + y 3 = 2 3 The best upper bound is when 5y 1 + 10y 2 + 7y 3 + 5y 4 is minimized! Chandra & Ruta (UIUC) CS473 9 Fall 2016 9 / 39

Dual LP: Example Thus, the optimum value of program maximize 4x 1 + 2x 2 subject to x 1 + 3x 2 5 2x 1 4x 2 10 x 1 + x 2 7 x 1 5 is upper bounded by the optimal value of the program minimize 5y 1 + 10y 2 + 7y 3 + 5y 4 subject to y 1 + 2y 2 + y 3 + y 4 = 4 3y 1 4y 2 + y 3 = 2 y 1, y 2 0 Chandra & Ruta (UIUC) CS473 10 Fall 2016 10 / 39

Dual Linear Program Given a linear program Π in canonical form maximize subject to d j=1 c jx j d j=1 a ijx j b i i = 1, 2,... n the dual Dual(Π) is given by minimize subject to n i =1 b i y i n i =1 y i a ij = c j y i 0 j = 1, 2,... d i = 1, 2,... n Chandra & Ruta (UIUC) CS473 11 Fall 2016 11 / 39

Duality Theorems Theorem (Weak Duality) If x is a feasible solution to Π and y is a feasible solution to Dual(Π) then c x y b. Chandra & Ruta (UIUC) CS473 12 Fall 2016 12 / 39

Duality Theorems Theorem (Weak Duality) If x is a feasible solution to Π and y is a feasible solution to Dual(Π) then c x y b. Theorem (Strong Duality) If x is an optimal solution to Π and y is an optimal solution to Dual(Π) then c x = y b. Many applications! Maxflow-Mincut theorem can be deduced from duality. Chandra & Ruta (UIUC) CS473 12 Fall 2016 12 / 39

Proof of Weak Duality We already saw the proof by the way we derived it but we will do it again formally. Since y is feasible to Dual(Π): y A = c Therefore c x = y Ax Since x is feasible Ax b and hence, c x = y Ax y b Chandra & Ruta (UIUC) CS473 13 Fall 2016 13 / 39

Duality for another canonical form maximize 4x 1 + x 2 + 3x 3 subject to x 1 + 4x 2 2 2x 1 x 2 + x 3 4 x 1, x 2, x 3 0 Chandra & Ruta (UIUC) CS473 14 Fall 2016 14 / 39

Duality for another canonical form maximize 4x 1 + x 2 + 3x 3 subject to x 1 + 4x 2 2 2x 1 x 2 + x 3 4 x 1, x 2, x 3 0 Choose non-negative y 1, y 2 and multiply inequalities maximize 4x 1 + x 2 + 3x 3 subject to y 1 (x 1 + 4x 2 ) 2y 1 y 2 (2x 1 x 2 + x 3 ) 4y 2 x 1, x 2, x 3 0 Chandra & Ruta (UIUC) CS473 14 Fall 2016 14 / 39

Duality for another canonical form Choose non-negative y 1, y 2 and multiply inequalities maximize 4x 1 + x 2 + 3x 3 subject to y 1 (x 1 + 4x 2 ) 2y 1 y 2 (2x 1 x 2 + x 3 ) 4y 2 x 1, x 2, x 3 0 Adding the inequalities we get an inequality below that is valid for any feasible x and any non-negative y: (y 1 + 2y 2 )x 1 + (4y 1 y 2 )x 2 + y 2 2y 1 + 4y 2 Suppose we choose y 1, y 2 such that y 1 + 2y 2 4 and 4y 2 y 2 1 and 2y 1 3 Then, since x 1, x 2, x 3 0, we have 4x 1 + x 2 + 3x 3 2y 1 + 4y 2 Chandra & Ruta (UIUC) CS473 15 Fall 2016 15 / 39

Duality for another canonical form is upper bounded by maximize 4x 1 + x 2 + 3x 3 subject to x 1 + 4x 2 2 2x 1 x 2 + x 3 4 x 1, x 2, x 3 0 minimize 2y 1 + 4y 2 subject to y 1 + 2y 2 4 4y 1 y 2 1 2y 1 3 y 1, y 2 0 Chandra & Ruta (UIUC) CS473 16 Fall 2016 16 / 39

Duality for another canonical form Compactly, For the primal LP max cx subject to Ax b, x 0 the dual LP is min yb subject to ya c, y 0 Chandra & Ruta (UIUC) CS473 17 Fall 2016 17 / 39

Some Useful Duality Properties Assume primal LP is a maximization LP. For a given LP, Dual is another LP. The variables in the dual correspond to non-trival primal constraints and vice-versa. Dual of the dual LP give us back the primal LP. Weak and strong duality theorems. If primal is unbounded (objective achieves infinity) then dual LP is infeasible. Why? If dual LP had a feasible solution it would upper bound the primal LP which is not possible. If primal is infeasible then dual LP is unbounded. Primal and dual optimum solutions satisfy complementary slackness conditions (discussed soon). Chandra & Ruta (UIUC) CS473 18 Fall 2016 18 / 39

Part II Examples of Duality Chandra & Ruta (UIUC) CS473 19 Fall 2016 19 / 39

Max matching in bipartite graph as LP Input:G = (V = L R, E) max s.t. uv E x uv x uv 1 v V. uv E x uv 0 uv E Chandra & Ruta (UIUC) CS473 20 Fall 2016 20 / 39

Network flow s-t flow in directed graph G = (V, E) with capacities c. Assume for simplicity that no incoming edges into s. max x(s, v) (s,v) E x(u, v) x(v, w) = 0 v V \ {s, t} (u,v) E (v,w) E x(u, v) c(u, v) (u, v) E x(u, v) 0 (u, v) E. Chandra & Ruta (UIUC) CS473 21 Fall 2016 21 / 39

Dual of Network Flow Chandra & Ruta (UIUC) CS473 22 Fall 2016 22 / 39

Part III Farkas Lemma and Strong Duality Chandra & Ruta (UIUC) CS473 23 Fall 2016 23 / 39

Optimization vs Feasibility Suppose we want to solve LP of the form: max cx subject to Ax b It is an optimization problem. Can we reduce it to a decision problem? Chandra & Ruta (UIUC) CS473 24 Fall 2016 24 / 39

Optimization vs Feasibility Suppose we want to solve LP of the form: max cx subject to Ax b It is an optimization problem. Can we reduce it to a decision problem? Yes, via binary search. Find the largest values of σ such that the system of inequalities Ax b, cx σ is feasible. Feasible implies that there is at least one solution. Caveat: to do binary search need to know the range of numbers. Skip for now since we need to worry about precision issues etc. Chandra & Ruta (UIUC) CS473 24 Fall 2016 24 / 39

Certificate for (in)feasibility Suppose we have a system of m inequalities in n variables defined by Ax b How can we convince some one that there is a feasible solution? How can we convince some one that there is no feasible solution? Chandra & Ruta (UIUC) CS473 25 Fall 2016 25 / 39

Theorem of the Alternatives Theorem Let A R m n and b R m. The system Ax b is either feasible of if it is infeasible then there is a y R m such that y 0 and ya = 0 and yb < 0. In other words, if Ax b is infeasible we can demonstrate it via the following compact contradiction. Find a non-negative combination of the rows of A (given by certificate y) to derive 0 < 1. The preceding theorem can be used to prove strong duality. A fair amount of formal detail though geometric intuition is reasonable. Chandra & Ruta (UIUC) CS473 26 Fall 2016 26 / 39

Farkas Lemma From the theorem of alternatives we can derive a useful version of Farkas lemma. Theorem A system Ax = b, x 0 is either feasible or there is a y such that ya 0 and yb < 0. Then the following hold: Nice geometric interpretation. Chandra & Ruta (UIUC) CS473 27 Fall 2016 27 / 39

Complementary Slackness Theorem Let x be any optimum solution to primal LP Π in the canonical form max cx, Ax b, x 0 and y be an optimum solution to the dual LP Dual(Π) which is min yb, ya c, y 0. Then the following hold. If y i > 0 then n j=1 a ijx j = b j (the primal constraint for row i is tight). If x j > 0 then m i =1 y i a ij = c j (the dual constraint for row j is tight). The converse also hold: if x and y are primal and dual feasible and satisfy complementary slackness conditions then both must be optimal. Very useful in various applications. Nice geometric interpretation. Chandra & Ruta (UIUC) CS473 28 Fall 2016 28 / 39

Part IV Integer Linear Programming Chandra & Ruta (UIUC) CS473 29 Fall 2016 29 / 39

Integer Linear Programming Problem Find a vector x Z d (integer values) that maximize subject to d j=1 c jx j d j=1 a ijx j b i for i = 1... n Input is matrix A = (a ij ) R n d, column vector b = (b i ) R n, and row vector c = (c j ) R d Chandra & Ruta (UIUC) CS473 30 Fall 2016 30 / 39

Factory Example maximize x 1 + 6x 2 subject to x 1 200 x 2 300 x 1 + x 2 400 x 1, x 2 0 Suppose we want x 1, x 2 to be integer valued. Chandra & Ruta (UIUC) CS473 31 Fall 2016 31 / 39

Factory Example Figure x 2 300 x 1 200 1 Feasible values of x 1 and x 2 are integer points in shaded region 2 Optimization function is a line; moving the line until it just leaves the final integer point in feasible region, gives optimal values Chandra & Ruta (UIUC) CS473 32 Fall 2016 32 / 39

Integer Programming Can model many difficult discrete optimization problems as integer programs! Therefore integer programming is a hard problem. NP-hard. Can relax integer program to linear program and approximate. Practice: integer programs are solved by a variety of methods 1 branch and bound 2 branch and cut 3 adding cutting planes 4 linear programming plays a fundamental role Chandra & Ruta (UIUC) CS473 33 Fall 2016 33 / 39

Example: Maximum Independent Set Definition Given undirected graph G = (V, E) a subset of nodes S V is an independent set (also called a stable set) if for there are no edges between nodes in S. That is, if u, v S then (u, v) E. Input Graph G = (V, E) Goal Find maximum sized independent set in G Chandra & Ruta (UIUC) CS473 34 Fall 2016 34 / 39

Example: Dominating Set Definition Given undirected graph G = (V, E) a subset of nodes S V is a dominating set if for all v V, either v S or a neighbor of v is in S. Input Graph G = (V, E), weights w(v) 0 for v V Goal Find minimum weight dominating set in G Chandra & Ruta (UIUC) CS473 35 Fall 2016 35 / 39

Example: s-t minimum cut and implicit constraints Input Graph G = (V, E), edge capacities c(e), e E. s, t V Goal Find minimum capacity s-t cut in G. Chandra & Ruta (UIUC) CS473 36 Fall 2016 36 / 39

Linear Programs with Integer Vertices Suppose we know that for a linear program all vertices have integer coordinates. Chandra & Ruta (UIUC) CS473 37 Fall 2016 37 / 39

Linear Programs with Integer Vertices Suppose we know that for a linear program all vertices have integer coordinates. Then solving linear program is same as solving integer program. We know how to solve linear programs efficiently (polynomial time) and hence we get an integer solution for free! Luck or Structure: 1 Linear program for flows with integer capacities have integer vertices 2 Linear program for matchings in bipartite graphs have integer vertices 3 A complicated linear program for matchings in general graphs have integer vertices. All of above problems can hence be solved efficiently. Chandra & Ruta (UIUC) CS473 37 Fall 2016 37 / 39

Linear Programs with Integer Vertices Meta Theorem: A combinatorial optimization problem can be solved efficiently if and only if there is a linear program for problem with integer vertices. Consequence of the Ellipsoid method for solving linear programming. In a sense linear programming and other geometric generalizations such as convex programming are the most general problems that we can solve efficiently. Chandra & Ruta (UIUC) CS473 38 Fall 2016 38 / 39

Summary 1 Linear Programming is a useful and powerful (modeling) problem. 2 Can be solved in polynomial time. Practical solvers available commercially as well as in open source. Whether there is a strongly polynomial time algorithm is a major open problem. 3 Geometry and linear algebra are important to understand the structure of LP and in algorithm design. Vertex solutions imply that LPs have poly-sized optimum solutions. This implies that LP is in NP. 4 Duality is a critical tool in the theory of linear programming. Duality implies the Linear Programming is in co-np. Do you see why? 5 Integer Programming in NP-Complete. LP-based techniques critical in heuristically solving integer programs. Chandra & Ruta (UIUC) CS473 39 Fall 2016 39 / 39