CS 50 - Algorithms and Complexity Linear Programming, the Simplex Method, and Hard Problems Sean Anderson 2/15/18 Portland State University
Table of contents 1. The Simplex Method 2. The Graph Problem to End all (of our) Graph Problems. Dynamic Programming and Time/Space Trade-offs 1
The Simplex Method
Linear Programming Revisited Programming = Planning Systems of n variables, m constraints of form: a 1 x 1 + + a n x n / / = b 2
A Touch of Linear Algebra I m breaking my promise not to do much linear algebra, but don t worry, we don t need much. A few key terms: Linear Combination: a vector Y is a linear combination of vectors X 1, X 2 if a 1 X 1 + + a i X i = Y Linear Independence: a set of vectors is linearly independent if none of its elements are a linear combination of the others Basis: a set of linearly independent vectors in a vector space such that all other vectors are a linear combination of them
Word Problem The Brewer s Problem: suppose an amateur brewer has 4 units of wheat and 6 units of hops. Brewing a normal beer requires 1 unit of each; brewing a ridiculous IPA requires 1 unit of wheat and of hops. But an IPA sells for $5 and a regular beer only sells for $. How much of each should they make? Optimize: z = b + 5a Where: 0 b + a 4 0 b + a 6 4
Geometric This is equivalent to finding the closest point to an m-dimensional hyperplane on the surface of a convex region 5
Vertices Provably, the optimal solution is at a vertex 6
Profit Brewery problem: objective function (0, 2) (12, 28) 1A + 2B = $1600 (26, 14) Beer 1A + 2B = $800 (0, 0) Ale (4, 0) 1A + 2B = $442 16 Source: Kleinberg and Tardos Slides 7
Standard Form The simplex method requires our equations to be actual equations, not inequalities, and a few other things Some changes: Must be maximization problem - if minimization, negate all weights All variables must be positive - negative/free x becomes x 1 x 2 All constraints must be equalities - add slack variables Optimize: z = b + 5a Where: b + a + u = 4 b + a + v = 6 8
Strategy and Tableaus Strategy: find a Basic Feasible Solution - a solution with respect to a given basis The basis is m independent variables, and the solution assumes we maximize them and compute the rest accordingly We build a tableau to track the equations and update with new, adjacent bases 9
Simplex Tableau b a u v Solution u 1 1 1 0 4 v 1 0 1 6 - -5 0 0 0 Translation: our proposed solution is that u = 4 and v = 6. This is a Basic Feasible Solution for the basis of (u, v), and represents a vertex where b, a = 0. Bottom row: negative values represent the weights of the variables we could increase, and the last column represents the objective value of this solution. Increasing b to 6 would yield z = 12, and a to 2 would yield z = 10. These are the two neighboring vertices. 10
Rebasing a has the highest negative weight in the objective row, so let s make it basic. We need to choose one of u or v to make non-basic. We choose by the θ ratio: old value / new variable s entry θ u = 4 1 ; θ v = 6 So we will keep u basic and remove v 11
Tableau 2 Need to normalize a in its row: b a u v Solution u 1 1 1 0 4 1 1 a 1 0 2 - -5 0 0 0 Then substitute for a in the other equations (subtract each entry*a s value in the other equation): b a u v Solution u 1 1 1-1 1-0 0-1 4-2 1 1 a 1 0 2 ( 5 1 ) 5 ( 5 1) 0 ( 5 1 ) -5*-2 12
Tableau 2 b a u v Solution 2 u 0 1 1 2 1 1 a 1 0 2 4 5 0 2 10 So now we have a solution when u and a are both 2. But we can see that b still has more to contribute! θ u = 2 2 = ; θ b = 2 1 = 6 So repeat again, replacing u with b as a basis. 1
Tableau b a u v Solution b 1 0 2 1 2 1 a 1 1 0 - ( 2 )( 1 ) 1 ( 1 2 )( 1 ) 2 1 4 ( 4 ) 0 0 ( 2 )( 4 ) 5 ( 1 2 )( 4 ) 10 ( 4 ) b a u v Solution b 1 0 2 1 2 a 0 1 1 1 2 2 1 0 0 2 1 14 All variables in solution are non-negative, so this is an optimal solution! b =, a = 1, u = 0, v = 0: z = 14 Note that sometimes our slack variables might be positive still, but in this case they aren t - we re efficient with our materials. 14
Simplex Procedure 1. Initialize a tableau with the problem in standard form and solution row 2. Check if solution row is all positive; if it is, stop - optimal solution. If not, choose a variable to make basic (entering variable) - probably the one with the lowest value in the solution row. Mark it as the pivot column. 4. Choose a basic variable to make non-basic (departing variable) - the one with the smallest θ ratio. Mark its row as the pivot row. 5. Form the new tableau by: Divide the elements of the pivot row by its element in the pivot column Subtract from each other row (its value in the pivot column)*(the pivot row) (This is similar to Gaussian elimination; it represents solving for one variable in terms of the others and substituting) 15
Simplex Analysis In some degenerate cases, multiple bases might represent the same vertex - this could lead to infinite loops! Some variations resolve θ ties systematically to avoid repeats, or other ways of fixing these. Still, worst case performance is exponential - systematic walk of O(m n ) vertices! In practice, most problems run in O(mn) time. Another algorithm, the Ellipsoid Method, is guaranteed (weak) polynomial time but in practice is much slower. Interior point method is also polynomial and actually competitive in practice. 16
The Graph Problem to End all (of our) Graph Problems
Traveling Salesman Traveling Salesman Instance: a connected weighted graph G Solution: a minimum Hamiltonian circuit of G - a cycle that visits every node exactly once 17
Brute Forcing Traveling Salesman In the worst case, an n-clique, every ordering of vertices is a Hamiltonian path So brute force is O( V!) - permutations of V 18
Discrete Knapsack {0,1} Knapsack Instance: a set of n items with values {v 1,, v n } and weights {w 1,, w n }, and a capacity W Solution: a vector X {0, 1} n that maximizes v 1 x 1 + + v n x n constrained by w 1 x 1 + + w n x n W 19
Brute Forcing Discrete Knapsack A solution is n independent binary decisions, so a brute force method is O(2 n ) 20
Integer Linear Programming Traveling Salesman can be thought of as a system of linear equations, if we think of each variable x i {0, 1} representing whether edge i is in the path. Like the {0,1} Knapsack Problem, it can be consider Integer Linear Programming or just Integer Programming. Integer Programming is HARD - we can no longer guarantee that our optimum is at a vertex, because the vertices of the constrained system may not be at integers! 21
Branch-and-Bound Idea: search the solution space using the (continuous) linear programming to guide us to upper and lower bounds. Build a search tree of candidates and prune aggressively. Notation: U i (x) and L i (x) are upper and lower bounds of variable v at node i 22
Branch-and-Bound Procedure Basic Procedure Without Pruning 1. Find optimal relaxed linear solution; round all variables down and find z 2. Root node: U i relaxed optimum; L i rounded. Select variable x M with greatest fractional part 4. Create child nodes with additional constraints: x M U i (x M ) x M L i (x M ) 5. Recurse on child nodes Find maximum non-fractional node Use α β pruning from AI 2
BnB Analysis Tree search: O(2 n ) worst case Still better than the O(n!) of a brute force approach 24
Dynamic Programming and Time/Space Trade-offs
Next Week Another approach to hard problems: writing things down! 25
References i