Linear programming Linear programming. Optimize a linear function subject to linear inequalities. (P) max c j x j n j= n s. t. a ij x j = b i i m j= x j 0 j n (P) max c T x s. t. Ax = b Lecture slides by Kevin Wayne Last updated on 7/25/7 :09 AM 2 Linear programming Linear programming. Optimize a linear function subject to linear inequalities. Generalizes: Ax = b, 2-person zero-sum games, shortest path, max flow, assignment problem, matching, multicommodity flow, MST, min weighted arborescence, Why significant? Design poly-time algorithms. Design approximation algorithms. Solve NP-hard problems using branch-and-cut. Ranked among most important scientific advances of 20th century. Reference: The Allocation of Resources by Linear Programming, Scientific American, by Bob Bland
Brewery problem Small brewery produces ale and beer. Production limited by scarce resources: corn, hops, barley malt. Recipes for ale and beer require different proportions of resources. Brewery problem objective function Beverage Corn (pounds) Hops (ounces) Malt (pounds) Ale (barrel) 5 4 5 Beer (barrel) 5 4 20 constraint 480 60 90 Profit () 2 Ale Beer max A + 2B s. t. 5A + 5B 480 4A + 4B 60 5A + 20B 90 A, B 0 Profit Corn Hops Malt How can brewer maximize profits? Devote all resources to ale: 4 barrels of ale 442 Devote all resources to beer: 2 barrels of beer 76 7.5 barrels of ale, 29.5 barrels of beer 776 2 barrels of ale, 28 barrels of beer 800 5 constraint decision variable 6 Reference 7
Standard form of a linear program Brewery problem: converting to standard form Standard form of an LP. Input: real numbers a ij, c j, b i. Output: real numbers x j. n = # decision variables, m = # constraints. Maximize linear objective function subject to linear inequalities. Original input. max A + 2B s. t. 5A + 5B 480 4A + 4B 60 5A + 20B 90 A, B 0 (P) max c j x j n j= n s. t. a ij x j = b i i m j= x j 0 j n (P) max c T x s. t. Ax = b Standard form. Add slack variable for each inequality. Now a 5-dimensional problem. Linear. No x 2, x y, arccos(x), etc. Programming. Planning (term predates computer programming). max A + 2B s. t. 5A + 5B + S C = 480 4A + 4B + S H = 60 5A + 20B + S M = 90 9 0 Equivalent forms Easy to convert variants to standard form. (P) max c T x s. t. Ax = b Less than to equality. x + 2y z 7 x + 2y z + s = 7, s 0 Greater than to equality. x + 2y z 7 x + 2y z s = 7, s 0 Min to max. min x + 2y z max x 2y + z Unrestricted to nonnegative. x unrestricted x = x + x, x + 0, x 0
Profit Fundamental questions LP. For A R m n, b R m, c R n, α R, does there exist x R n such that: Ax = b,, c T x α? Q. Is LP in NP? Q. Is LP in co-np? Q. Is LP in P? Q. Is LP in P R? Blum Shub Smale model Input size. n = number of variables. m = number of constraints. L = number of bits to encode input. Brewery problem: feasible region Brewery problem: objective function Hops 4A + 4B 60 Malt 5A + 20B 90 (0, 2) (0, 2) (2, 28) (2, 28) (26, 4) Corn 5A + 5B 480 (26, 4) A + 2B = 600 Beer Beer A + 2B = 800 (0, 0) Ale (4, 0) 5 (0, 0) Ale (4, 0) A + 2B = 442 6
Brewery problem: geometry Convexity Brewery problem observation. Regardless of objective function coefficients, an optimal solution occurs at a vertex. Convex set. If two points x and y are in the set, then so is λ x + ( λ) y for 0 λ. convex combination not a vertex iff d 0 s.t. x ± d in set Vertex. A point x in the set that can t be written as a strict convex combination of two distinct points in the set. (0, 2) vertex (2, 28) vertex x (26, 4) y Beer convex not convex (0, 0) Ale (4, 0) Observation. LP feasible region is a convex set. 7 8 Purificaiton Purificaiton Theorem. If there exists an optimal solution to (P), then there exists one that is a vertex. Theorem. If there exists an optimal solution to (P), then there exists one that is a vertex. (P) max c T x s. t. Ax = b Intuition. If x is not a vertex, move in a non-decreasing direction until you reach a boundary. Repeat. Pf. Suppose x is an optimal solution that is not a vertex. There exist direction d 0 such that x ± d P. A d = 0 because A(x ± d) = b. Assume c T d 0 (by taking either d or d). Consider x + λ d, λ > 0 : x - d x + d x xʹ = x + α * d Case. [ there exists j such that d j < 0 ] Increase λ to λ * until first new component of x + λ d hits 0. x + λ * d is feasible since A(x + λ * d) = Ax = b and x + λ * y 0. x + λ * d has one more zero component than x. c T xʹ = c T (x + λ * d) = c T x + λ * c T d c T x. d k = 0 whenever x k = 0 because x ± d P 9 20
Purificaiton Theorem. If there exists an optimal solution to (P), then there exists one that is a vertex. Pf. Suppose x is an optimal solution that is not a vertex. There exist direction d 0 such that x ± d P. A d = 0 because A(x ± d) = b. Assume c T d 0 (by taking either d or d). Consider x + λ d, λ > 0 : Case 2. [d j 0 for all j ] x + λd is feasible for all λ 0 since A(x + λd) = b and x + λd. As λ, c T (x + λd) because c T d < 0. if c T d = 0, choose d so that case applies 2 Intuition Basic feasible solution Intuition. A vertex in R m is uniquely specified by m linearly independent equations. Theorem. Let P = { x : Ax = b, }. For x P, define B = { j : x j > 0 }. Then, x is a vertex iff A B has linearly independent columns. 4A + 4B 60 5A + 20B 90 Notation. Let B = set of column indices. Define A B to be the subset of columns of A indexed by B. (26, 4) 4A + 4B = 60 5A + 20B = 90 Ex. " 2 0% " 7 % A = 7 2, b = 6 # 0 0 0 5& # 0 & " 2 % " 2 % 0 x =, B = {, }, A B = 7 2 # 0 # 0 0& & 2 24
Basic feasible solution Basic feasible solution Theorem. Let P = { x : Ax = b, }. For x P, define B = { j : x j > 0 }. Then, x is a vertex iff A B has linearly independent columns. Theorem. Let P = { x : Ax = b, }. For x P, define B = { j : x j > 0 }. Then, x is a vertex iff A B has linearly independent columns. Pf. Assume x is not a vertex. There exist direction d 0 such that x ± d P. A d = 0 because A(x ± d) = b. Define Bʹ = { j : d j 0 }. A Bʹ has linearly dependent columns since d 0. Moreover, d j = 0 whenever x j = 0 because x ± d 0. Thus Bʹ B, so A Bʹ is a submatrix of A B. Therefore, A B has linearly dependent columns. Pf. Assume A B has linearly dependent columns. There exist d 0 such that A B d = 0. Extend d to R n by adding 0 components. Now, A d = 0 and d j = 0 whenever x j = 0. For sufficiently small λ, x ± λ d P x is not a vertex. 25 26 Basic feasible solution Basic feasible solution: example Theorem. Given P = { x : Ax = b, }, x is a vertex iff there exists B {,, n } such B = m and: A B is nonsingular. x B = A B - b 0. x N = 0. basic feasible solution Pf. Augment A B with linearly independent columns (if needed). " 2 0% " 7 % A = 7 2, b = 6 # 0 0 0 5& # 0 & " 2 % " 2 0 % 0 x =, B = {,, 4 }, A B = 7 2 # 0 # 0 0 5 & & Basic feasible solutions. max A + 2B s. t. 5A + 5B + S C = 480 4A + 4B + S H = 60 5A + 20B + S M = 90 {B, S H, S M } (0, 2) Beer {S H, S M, S C } (0, 0) Basis {A, B, S M } (2, 28) Ale Infeasible {A, B, S H } (9.4, 25.5) {A, B, S C } (26, 4) {A, S H, S C } (4, 0) Assumption. A R m n has full row rank. 27 28
Fundamental questions LP. For A R m n, b R m, c R n, α R, does there exist x R n such that: Ax = b,, c T x α? Q. Is LP in NP? A. Yes. Number of vertices ( ) nm. C(n, m) = n m Cramer s rule can check a vertex in poly-time. Cramer s rule. For B R n n invertible, b R n, the solution to Bx = b is given by: x i = det(b i ) det(b) replace ith column of B with b 29 Simplex algorithm: intuition Simplex algorithm: initialization Simplex algorithm. [George Dantzig 947] Move from BFS to adjacent BFS, without decreasing objective function. replace one basic variable with another A + 2B Z = 0 5A + 5B + S C = 480 4A + 4B + S H = 60 5A + 20B + S M = 90 Basis = {S C, S H, S M } A = B = 0 Z = 0 S C = 480 S H = 60 S M = 90 edge Greedy property. BFS optimal iff no adjacent BFS is better. Challenge. Number of BFS can be exponential! 2
Simplex algorithm: pivot Simplex algorithm: pivot A + 2B Z = 0 5A + 5B + S C = 480 4A + 4B + S H = 60 5A + 20B + S M = 90 Basis = {S C, S H, S M } A = B = 0 Z = 0 S C = 480 S H = 60 S M = 90 A + 2B Z = 0 5A + 5B + S C = 480 4A + 4B + S H = 60 5A + 20B + S M = 90 Basis = {S C, S H, S M } A = B = 0 Z = 0 S C = 480 S H = 60 S M = 90 Substitute: B = /5 (480 5A S C ) 6 A 5 2 S C Z = 76 A + B + S 5 C = 2 8 A 4 S 5 C + S H = 2 85 A 4 S C + S M = 550 Basis = {B, S H, S M } A = S C = 0 Z = 76 B = 2 S H = 2 S M = 550 Q. Why pivot on column 2 (or )? A. Each unit increase in B increases objective value by 2. Q. Why pivot on row 2? A. Preserves feasibility by ensuring RHS 0. min ratio rule: min { 480/5, 60/4, 90/20 } 4 Simplex algorithm: pivot 2 Simplex algorithm: optimality 6 A 2 5 S C Z = 76 A + B + 5 S C = 2 8 A 4 5 S C + S H = 2 85 A 4 S C + S M = 550 Substitute: A = /8 (2 + 4/5 S C S H ) Basis = {B, S H, S M } A = S C = 0 Z = 76 B = 2 S H = 2 S M = 550 Q. When to stop pivoting? A. When all coefficients in top row are nonpositive. Q. Why is resulting solution optimal? A. Any feasible solution satisfies system of equations in tableaux. In particular: Z = 800 S C 2 S H, S C 0, S H 0. Thus, optimal objective value Z* 800. Current BFS has value 800 optimal. S C 2 S H Z = 800 B + 0 S C + S 8 H = 28 A 0 S C + 8 S H = 2 25 S 6 C 85 S 8 H + S M = 0 A, B, S C, S H, S M 0 Basis = {A, B, S M } S C = S H = 0 Z = 800 B = 28 A = 2 S M = 0 S C 2 S H Z = 800 B + 0 S C + S 8 H = 28 A 0 S C + 8 S H = 2 25 S 6 C 85 S 8 H + S M = 0 A, B, S C, S H, S M 0 Basis = {A, B, S M } S C = S H = 0 Z = 800 B = 28 A = 2 S M = 0 5 6
Simplex tableaux: matrix form Simplex algorithm: corner cases Initial simplex tableaux. c T B x B + c T N x N = Z A B x B + A N x N = b x B, x N 0 Simplex algorithm. Missing details for corner cases. Q. What if min ratio test fails? Q. How to find initial basis? Q. How to guarantee termination? Simplex tableaux corresponding to basis B. (c N T c B T A B A N ) x N = Z c B T A B b I x B + A B A N x N = A B b x B, x N 0 subtract c B T A B - times constraints multiply by A B - x B = A B - b 0 x N = 0 basic feasible solution c N T c B T A B - A N 0 optimal basis 7 8 Unboundedness Phase I simplex method Q. What happens if min ratio test fails? all coefficients in entering column are nonpositive Q. How to find initial basis? (P) max c T x s. t. Ax = b + 2x 4 + 20x 5 Z = 2 x 4x 4 8x 5 = x 2 + 5x 4 2x 5 = 4 x = 5 x, x 2, x, x 4, x 5 0 A. Solve (P ), starting from basis consisting of all the z i variables. m ( P ") max z i i= s. t. A x + I z = b x, z 0 A. Unbounded objective function. Z = 2 + 20x 5 " # x x 2 x x 4 x 5 % & = " + 8x 5 % 4 + 2x 5 5 # 0 0 & Case : min > 0 (P) is infeasible. Case 2: min = 0, basis has no z i variables okay to start Phase II. Case a: min = 0, basis has z i variables. Pivot z i variables out of basis. If successful, start Phase II; else remove linear dependent rows. 9 40
Simplex algorithm: degeneracy Simplex algorithm: degeneracy Degeneracy. New basis, same vertex. Degeneracy. New basis, same vertex. Degenerate pivot. Min ratio = 0. x 4 4 20x 5 + x 2 6 6x 7 Z = 0 x + x 4 4 8x 5 x 6 + 9x 7 = 0 x 2 + 2 x 4 2x 5 2 x 6 + x 7 = 0 x + x 6 = x, x 2, x, x 4, x 5, x 6, x 7 0 Cycling. Infinite loop by cycling through different bases that all correspond to same vertex. Anti-cycling rules. Bland s rule: choose eligible variable with smallest index. Random rule: choose eligible variable uniformly at random. Lexicographic rule: perturb constraints so nondegenerate. 4 42 Lexicographic rule Lexicographic rule Intuition. No degeneracy no cycling. Intuition. No degeneracy no cycling. Perturbed problem. much much greater, say ε i = δ i for small δ Perturbed problem. much much greater, say ε i = δ i for small δ ( P ") max c T x s. t. Ax = b + ε # ε & % ( ε where ε = % 2 (, such that ε ε 2 # ε n %! ( % ( ε n ( P ") max c T x s. t. Ax = b + ε # ε & % ( ε where ε = % 2 (, such that ε ε 2 # ε n %! ( % ( ε n Lexicographic rule. Apply perturbation virtually by manipulating ε symbolically: 7 + 5ε + ε 2 + 8ε 7 + 5ε + 4ε 2 + ε Claim. In perturbed problem, x B = A - B (b + ε) is always nonzero. Pf. The j th component of x B is a (nonzero) linear combination of the components of b + ε contains at least one of the ε i terms. Corollary. No cycling. which can t cancel 4 44
Simplex algorithm: practice Remarkable property. In practice, simplex algorithm typically terminates after at most 2(m + n) pivots. but no polynomial pivot rule known Issues. Avoid stalling. Choose the pivot. Maintain sparsity. Ensure numerical stability. Preprocess to eliminate variables and constraints. Commercial solvers can solve LPs with millions of variables and tens of thousands of constraints. 45