Linear Programming Linear Programming Lecture Linear programming. Optimize a linear function subject to linear inequalities. (P) max " c j x j n j= n s. t. " a ij x j = b i # i # m j= x j 0 # j # n (P) max c T x s. t. Ax = b x " 0 Kevin Wayne Computer Science Department Princeton University COS 52 Fall 2007 2 Linear Programming Linear programming. Optimize a linear function subject to linear inequalities. Generalizes: Ax = b, 2-person zero-sum games, shortest path, max flow, assignment problem, matching, multicommodity flow, MST, min weighted arborescence, Why significant?! Design poly-time algorithms.! Design approximation algorithms.! Solve NP-hard problems using branch-and-cut. Linear Programming I Ranked among most important scientific advances of 20th century. Reference: The Allocation of Resources by Linear Programming, Scientific American, by Bob Bland
Brewery Problem Brewery Problem Small brewery produces ale and beer.! Production limited by scarce resources: corn, hops, barley malt.! Recipes for ale and beer require different proportions of resources. objective function Beverage Corn (pounds) Hops (ounces) Malt (pounds) Ale (barrel) 5 4 5 Beer (barrel) 5 4 20 constraint 480 60 90 Profit () 2 Ale Beer max A + 2B s. t. 5A + 5B " 480 4A + 4B " 60 5A + 20B " 90 A, B # 0 Profit Corn Hops Malt How can brewer maximize profits?! Devote all resources to ale: 4 barrels of ale! 442! Devote all resources to beer: 2 barrels of beer! 76! 7.5 barrels of ale, 29.5 barrels of beer! 776! 2 barrels of ale, 28 barrels of beer! 800 constraint decision variable 5 6 Linear Programming I Standard Form LP "Standard form" LP.! Input: real numbers a ij, c j, b i.! Output: real numbers x j.! n = # decision variables, m = # constraints.! Maximize linear objective function subject to linear inequalities. (P) max " c j x j n j= n s. t. " a ij x j = b i # i # m j= x j 0 # j # n (P) max c T x s. t. Ax = b x " 0 Linear. No x 2, x y, arccos(x), etc. Programming. Planning (term predates computer programming). 8
Brewery Problem: Converting to Standard Form Equivalent Forms Original input. max A + 2B s. t. 5A + 5B " 480 4A + 4B " 60 5A + 20B " 90 A, B # 0 Easy to convert variants to standard form. (P) max c T x s. t. Ax = b x " 0 Standard form.! Add slack variable for each inequality.! Now a 5-dimensional problem. max A + 2B s. t. 5A + 5B + = 480 4A + 4B + = 60 5A + 20B + = 90 A, B, " 0 Less than to equality. x + 2y z " 7! x + 2y z + s = 7, s # 0 Greater than to equality. x + 2y z # 7! x + 2y z s = 7, s # 0 Min to max. min x + 2y z! max x 2y + z Unrestricted to nonnegative. x unrestricted! x = x + x, x + # 0, x # 0 9 0 Linear Programming I Fundamental Questions LP. For A % m&n, b % m, c % n,! %, does there exist x % n such that: Ax = b, x # 0, c T x #? Q. Is LP in NP? Q. Is LP in co-np? Q. Is LP in P? Q. Is LP in P %? Blum-Shub-Smale model Input size.! n = number of variables.! m = number of constraints.! L = number of bits to encode input. 2
Profit Linear Programming I Brewery Problem: Feasible Region Hops 4A + 4B " 60 Malt 5A + 20B " 90 (0, 2) (2, 28) (26, 4) Corn 5A + 5B " 480 Beer (0, 0) Ale (4, 0) 4 Brewery Problem: Objective Function Brewery Problem: Geometry Brewery problem observation. Regardless of objective function coefficients, an optimal solution occurs at a vertex. (0, 2) (0, 2) (2, 28) (2, 28) A + 2B = 600 vertex (26, 4) (26, 4) Beer A + 2B = 800 Beer (0, 0) Ale (4, 0) A + 2B = 442 (0, 0) Ale (4, 0) 5 6
Convexity Purificaiton Convex set. If two points x and y are in the set, then so is ) x + (- )) y for 0 " ) ". convex combination Vertex. A point x in the set that cant be written as a strict convex combination of two distinct points in the set. not a vertex iff, d ( 0 s.t. x ± d in set Theorem. If there exists an optimal solution to (P), then there exists one that is a vertex. (P) max c T x s. t. Ax = b x " 0 vertex Intuition. If x is not a vertex, move in a non-decreasing direction until you reach a boundary. Repeat. x y convex Observation. LP feasible region is a convex set. not convex x - d x + d x x = x +! * d 7 8 Purificaiton Purificaiton Theorem. If there exists an optimal solution to (P), then there exists one that is a vertex. Pf.! Suppose x is an optimal solution that is not a vertex.! There exist direction d ( 0 such that x ± d P.! A d = 0 because A(x ± d) = b.! Assume c T d " 0 (by taking either d or d).! Consider x + ) d, ) > 0 : Theorem. If there exists an optimal solution to (P), then there exists one that is a vertex. Pf.! Suppose x is an optimal solution that is not a vertex.! There exist direction d ( 0 such that x ± d P.! A d = 0 because A(x ± d) = b.! Assume c T d " 0 (by taking either d or d).! Consider x + ) d, ) > 0 : Case. [ there exists j such that d j < 0 ]! Increase ) to ) * until first new component of x + ) d hits 0.! x + ) * d is feasible since A(x + ) * d) = Ax = b and x + ) * y # 0.! x + ) * d has one more zero component than x.! c T x = c T (x + ) * d) = c T x + ) * c T d " c T x. d k = 0 whenever x k = 0 because x ± d P Case 2. [d j # 0 for all j ]! x + )d is feasible for all ) # 0 since A(x + )d) = b and x + )d # x # 0.! As ) * +, c T (x + )d) * + because c T d < 0. if c T d = 0, choose d so that case applies 9 20
Linear Programming I Intuition Intuition. A vertex in % m is uniquely specified by m linearly independent equations. 4A + 4B " 60 5A + 20B " 90 (26, 4) 4A + 4B = 60 5A + 20B = 90 22 Basic Feasible Solution Basic Feasible Solution Theorem. Let P = { x : Ax = b, x # 0. For x P, define B = { j : x j > 0. Then x is a vertex iff A B has linearly independent columns. Notation. Let B = set of column indices. Define A B to be the subset of columns of A indexed by B. Ex. " 2 0% " 7 % A = 7 2, b = 6 # 0 0 0 5& # 0 & " 2 % " 2 % 0 x =, B = {,, A B = 7 2 # 0 # 0 0& & Theorem. Let P = { x : Ax = b, x # 0. For x P, define B = { j : x j > 0. Then x is a vertex iff A B has linearly independent columns. Pf. -! Assume x is not a vertex.! There exist direction d ( 0 such that x ± d P.! A d = 0 because A(x ± d) = b.! Define B = { j : d j ( 0.! A B has linearly dependent columns since d ( 0.! Moreover, d j = 0 whenever x j = 0 because x ± d # 0.! Thus B. B, so A B is a submatrix of A B.! Therefore, A B has linearly dependent columns. 2 24
Basic Feasible Solution Basic Feasible Solution Theorem. Let P = { x : Ax = b, x # 0. For x P, define B = { j : x j > 0. Then x is a vertex iff A B has linearly independent columns. Pf.!! Assume A B has linearly dependent columns.! There exist d ( 0 such that A B d = 0.! Extend d to % n by adding 0 components.! Now, A d = 0 and d j = 0 whenever x j = 0.! For sufficiently small ), x ± ) d P! x is not a vertex. Theorem. Given P = { x : Ax = b, x # 0, x is a vertex iff there exists B. {,, n such B = m and:! A B is nonsingular.! x B = A - B b # 0.! x N = 0. basic feasible solution Pf. Augment A B with linearly independent columns (if needed). " 2 0% " 7 % A = 7 2, b = 6 # 0 0 0 5& # 0 & " 2 % " 2 0 % 0 x =, B = {,, 4, A B = 7 2 # 0 # 0 0 5 & & Assumption. A % m"n has full row rank. 25 26 Basic Feasible Solution: Example Fundamental Questions Basic feasible solutions. LP. For A % m&n, b % m, c % n,! %, does there exist x % n such that: Ax = b, x # 0, c T x #? max A + 2B s. t. 5A + 5B + = 480 4A + 4B + = 60 5A + 20B + = 90 A, B, " 0 {B (0, 2) Beer Basis {A, B (2, 28) Infeasible {A, B (9.4, 25.5) {A, B, (26, 4) Q. Is LP in NP? A. Yes.! Number of vertices " C(n, m) = ( n m).! Cramers rule! can check a vertex in poly-time. {, (0, 0) Ale {A, (4, 0) Cramers rule. For B % n&n invertible, b % n, the solution to Bx = b is given by: x i = det(b i ) det(b) replace ith column of B with b 27 28
Linear Programming I Simplex Algorithm: Intuition Simplex algorithm. [George Dantzig 947] Move from BFS to adjacent BFS, without decreasing objective function. replace one basic variable with another edge Greedy property. BFS optimal iff no adjacent BFS is better. Challenge. Number of BFS can be exponential! 0 Simplex Algorithm: Initialization Simplex Algorithm: Pivot A + 2B " Z = 0 5A + 5B + = 480 4A + 4B + = 60 5A + 20B + = 90 A, B, # 0 Basis = { A = B = 0 Z = 0 = 480 = 60 = 90 A + 2B " Z = 0 5A + 5B + = 480 4A + 4B + = 60 5A + 20B + = 90 A, B, # 0 Basis = { A = B = 0 Z = 0 = 480 = 60 = 90 Substitute: B = /5 (480 5A ) 6 A " 5 2 " Z = "76 A + B + S 5 C = 2 8 A " 4 S 5 C + = 2 85 A " 4 S C + = 550 A, B, # 0 Basis = {B A = = 0 Z = 76 B = 2 = 2 = 550 2
Simplex Algorithm: Pivot Simplex Algorithm: Pivot 2 A + 2B " Z = 0 5A + 5B + = 480 4A + 4B + = 60 5A + 20B + = 90 A, B, # 0 Basis = { A = B = 0 Z = 0 = 480 = 60 = 90 6 A " 2 5 " Z = "76 A + B + 5 = 2 8 A " 4 5 + = 2 85 A " 4 + = 550 A, B, # 0 Basis = {B A = = 0 Z = 76 B = 2 = 2 = 550 Q. Why pivot on column 2 (or )? A. Each unit increase in B increases objective value by 2. Q. Why pivot on row 2? A. Preserves feasibility by ensuring RHS # 0. min ratio rule: min { 480/5, 60/4, 90/20 Substitute: A = /8 (2 + 4/5 ) " " 2 " Z = "800 B + S 0 C + S 8 H = 28 A " 0 + 8 = 2 " 25 S 6 C " 85 S 8 H + = 0 A, B, # 0 Basis = {A, B = = 0 Z = 800 B = 28 A = 2 = 0 4 Simplex Algorithm: Optimality Simplex Tableaux: Matrix Form Q. When to stop pivoting? A. When all coefficients in top row are nonpositive. Q. Why is resulting solution optimal? A. Any feasible solution satisfies system of equations in tableaux.! In particular: Z = 800 2, # 0 # 0.! Thus, optimal objective value Z* " 800.! Current BFS has value 800! optimal. " " 2 " Z = "800 B + 0 + 8 = 28 A " 0 C + 8 H = 2 " 25 6 C " 85 8 H + = 0 A, B, # 0 Basis = {A, B = = 0 Z = 800 B = 28 A = 2 = 0 Initial simplex tableaux. Simplex tableaux corresponding to basis B. (c N T c B T x B + c N T x N = Z A B x B + A N x N = b x B, x N " 0 " c B T A B " A N ) x N = Z " c B T A B " b I x B + A B " A N x N = A B " b x B, x N # 0 x B = A B - b # 0 x N = 0 basic feasible solution c N T c B T A B - A N " 0 optimal basis subtract c B T A B - times constraints multiply by A B - 5 6
Simplex Algorithm: Corner Cases Unboundedness Simplex algorithm. Missing details for corner cases. Q. What if min ratio test fails? Q. How to find initial basis? Q. How to guarantee termination? Q. What happens if min ratio test fails? + 2x 4 + 20x 5 " Z = 2 x " 4x 4 " 8x 5 = x 2 + 5x 4 " 2x 5 = 4 x = 5 x, x 2, x, x 4, x 5 # 0 all coefficients in entering column are nonpositive A. Unbounded objective function. Z = 2 + 20x 5 " # x x 2 x x 4 x 5 % & = " + 8x 5 % 4 + 2x 5 5 # 0 0 & 7 8 Phase I Simplex Simplex Algorithm: Degeneracy Q. How to find initial basis? (P) max c T x s. t. Ax = b x " 0 Degeneracy. New basis, same vertex. A. Solve (P), starting from basis consisting of all the z i variables. m ( P ") max # z i i= s. t. A x + I z = b x, z 0 Degenerate pivot. Min ratio = 0.! Case : min > 0! (P) is infeasible.! Case 2: min = 0, basis has no z i variables! OK to start Phase II.! Case a: min = 0, basis has z i variables. Pivot z i variables out of basis. If successful, start Phase II; else remove linear dependent rows. x 4 4 " 20x 5 + x 2 6 " 6x 7 " Z = 0 x + x 4 4 " 8x 5 " x 6 + 9x 7 = 0 x 2 + 2 x 4 " 2x 5 " 2 x 6 + x 7 = 0 x + x 6 = x, x 2, x, x 4, x 5, x 6, x 7 # 0 9 40
Simplex Algorithm: Degeneracy Lexicographic Rule Degeneracy. New basis, same vertex. Intuition. No degeneracy! no cycling. Perturbed problem. ( P ") max c T x s. t. Ax = b + # x 0 #" & % ( " where " = % 2 (, such that " f " 2 f L f " n % M ( % ( " n much much greater, say / i =!0 i for small 0 Cycling. Infinite loop by cycling through different bases that all correspond to same vertex. Anti-cycling rules.! Blands rule: choose eligible variable with smallest index.! Random rule: choose eligible variable uniformly at random.! Lexicographic rule: perturb constraints so nondegenerate. Lexicographic rule. Apply perturbation virtually by manipulating / symbolically: 7 + 5" + " 2 + 8" # 7 + 5" + 4" 2 + " 4 42 Lexicographic Rule Simplex Algorithm: Practice Intuition. No degeneracy! no cycling. Perturbed problem. ( P ") max c T x s. t. Ax = b + # x 0 Claim. In perturbed problem, x B = A B - (b + #) is always nonzero. Pf. The j th component of x B is a (nonzero) linear combination of the components of b + #! contains at least one of the # i terms. Corollary. No cycling. #" & % ( " where " = % 2 (, such that " f " 2 f L f " n % M ( % ( " n much much greater, say / i =!0 i for small 0 which cant cancel Remarkable property. In practice, simplex algorithm typically terminates after at most 2(m + n) pivots. Issues.! Avoid stalling.! Choose the pivot.! Maintain sparsity.! Ensure numerical stability. but no polynomial pivot rule known! Preprocess to eliminate variables and constraints. Commercial solvers can solve LPs with millions of variables and tens of thousands of constraints. 4 44