University of Colorado Denver, Fall 2013, Prof. Engau 1 Problem-Solving in Operations Research 2 Brief History of Linear Programming 3 Review of Basic Linear Algebra
Linear Programming - The Story About How It Began by George B. Dantzig, Source: Operations Research, Vol. 50, No. 1, pp. 42-47, 2002 Linear programming can be viewed as part of a great revolutionary development which has given mankind the ability to state general goals and to lay out a path of detailed decisions to take in order to best achieve its goals when faced with practical situations of great complexity. 1 What are models? ways to formulate real-world problems in detailed mathematical terms 2 What are algorithms? techniques for solving the models 3 What are computers/software? engines for executing the steps of algorithms
Problem-Solving and Decision-Making Flowchart Statistics Computation
LP History I (1665-1936) Mathematical Preliminaries 1665 Finding a Minimum Solution of a Function Newton 1788 Lagrangian Multipliers Lagrange 1823 Solution of Inequalities Fourier 1826 Solution of Linear Equations Gauss 1873 Solution of Equations in Nonnegative Variables Gordon 1896 Solution of Linear Equations as a Combination of Extreme Point Solutions Minkowski 1903 Solution of Inequality Systems Farkas 1915 Positive Solution to Linear Equations Stiemke 1936 Transposition Theorem and Linear Inequalities Motzkin
LP History II (1939-1951) LP and the Simplex Method 1939 Mathematical Methods of Organization and Production Kantorovich (Noble Prize for Economics 1975) 1941 Structure of the American Economy Leontief (NPE 1973) 1941 Transportation Problem Hitchcock 1944 Games and Economic Behavior von Neumann, Morgenstern 1947 Linear Programming Model Dantzig 1950 First Solution of the Transportation Problem on a Computer SEAC, National Bureau of Standards 1951 Maximization of a Linear Function of Variables Subject to Linear Inequalities (The Simplex Method) Dantzig 1951 First Computer-Based Simplex Algorithm SEAC/NBS 1951 Primal-Dual Linear Programs von Neumann, Dantzig, Tucker
LP History III (1951-) Linear Programming Extensions 1951 Nonlinear Programming Kuhn, Tucker, Frisch 1952 Commercial Applications Charnes, Cooper, Mellon 1954 Network Flow Theory Ford, Fulkerson, Hoffman 1955 Large-Scale Decomposition Dantzig, Wolfe, Benders 1955 Stochastic Programming Dantzig, Wets, Birge, Beale 1957 Dynamic Programming Bellman 1958 Integer Programming Gomory, Johnson, Balas 1962 Complementary Pivot Theory Cottle, Danzig, Lemke 1965 Goal Programming Charnes, Cooper 1971 Computational Complexity Cook, Karp, Klee, Minty 1979 Ellipsoid Method Shor, Khachian 1984 Interior Point Methods Karmarkar 1996 Semidefinite/Conic Programming Vandenberghe, Boyd
Optimization and Lagrangean Multipliers in Calculus Let f : R n R 1, g : R n R m, h : R n R k be twice cont. diff.: minimize f (x 1, x 2,..., x n ) subject to g i (x 1, x 2,..., x n ) 0 for i = 1,..., m h j (x 1, x 2,..., x n ) = 0 for j = 1,..., k. Let y 0 R m and z R k free and consider the Lagrangean: L(x, y, z) = f (x) m i=1 y ig i (x) k j=1 z jh j (x) First-Order Necessary Conditions (on Gradient L) y i x L(x, y, z ) = y L(x, y, z ) = z L(x, y, z ) = 0 0, g i (x ) 0, and y i g i (x ) = h j (x ) = 0 for all i and j Second-Order Sufficiency Conditions (on Hessian H) If H is positive (negative) definite, then x is a min (max).
Linear Systems of Equations or Inequalities - Notation Let A R m n be a matrix with m rows a i R n for i = 1,..., m: a 11 a 12 a 13... a 1n a i1 a 21 a 22 a 23... a 2n A = a 31 a 32 a 33... a 3n R m n a i2 a........ i = a i3.... Rn a m1 a m2...... a mn a in Let b R m and x R n be two column vectors: b 1 b 2 x 2 b = b 3 R m x = x 3 R n.. b m x n x 1
Solving Linear Systems of Equations I - Rank Criterion Consider the linear system Ax = b (m equations, n variables): a 11 x 1 + a 12 x 2 + a 13 x 3 +... + a 1n x n = b 1 a 21 x 1 + a 22 x 2 + a 23 x 3 +... + a 2n x n = b 2 a 31 x 1 + a 32 x 2 + a 33 x 3 +... + a 3n x n = b 3...... a m1 x 1 + a m2 x 2 + a m3 x 3 +... + a mn x n = b m Case 1: rank(a) < rank(a b) Case 2: rank(a) = rank(a b) = n Case 3: rank(a) = rank(a b) < n no solution unique solution infinitely many solutions Optimization makes sense only in the third case! (Why?)
Writing Linear Systems in Nonnegative Variables Without loss of generality, we can assume x is nonnegative: Ax = b and x 0. If x is free, we can use two auxiliary variables and write x = x + x where x + 0 and x 0. In practical applications, x 0 is often part of the model. Geometrical Interpretation: The set S = {x R n : Ax = b} is an affine subspace and C = {x R n : x 0} is a convex cone. A set C is a cone if αc C whenever c C and α 0. A set C is convex if αc + (1 α)d C whenever c C, d C, and 0 α 1. (A cone C is convex iff C + C C.)
Writing Linear Systems of Equations as Inequalities Each set {x : a T i x = b i } is a hyperplane (n 1 dim. manifold). Halfspaces are H + = { a T i x b i } and H = { a T i x b i }. The normal vector a i R n points orthogonally into H +. Polyhedral sets are finite intersections of halfspaces. Wlog, we can write linear systems of equations as inequalities: Ax = b Ax b and Ax b Similarly, we can write inequalities as equalities in nonnegative variables using auxiliary slack or excess variables ( residuals ): Ax b Ax + w = b and w 0 Ax b Ax w = b and w 0 Quiz: The variable x is still free - can you make it nonnegative?
Solving Linear Systems of Equations II - Geometry Exercise: Geometrically characterize the solution set of x 1 + 2x 2 + 3x 3 = 6 x 1 + x 2 + x 3 + x 4 = 4 x 1, x 2, x 3, x 4 0 One approach: Turn slacks x 1 and x 4 into inequalities: x 1 = 6 2x 2 3x 3 0 x 4 = 4 x 1 x 2 x 3 = 4 (6 2x 2 3x 3 ) x 2 x 3 = x 2 + 2x 3 2 0 With this dictionary draw a polyhedron in x 2 -x 3 coordinates: 2x 2 + 3x 3 6 x 2 + 2x 3 2 x 2, x 3 0 2 1 x 3 1 2 3 x 2
Solving Linear Systems III - Representation Theorem We can represent the solutions to linear systems as convex combinations of the extreme points of their polyhedral sets. 2 1 x is an extreme point of a polyhedral set S if x = y = z whenever x = αy + (1 α)z for y, z S and 0 α 1. Extreme point representation: For every x S, there is a vector of nonnegative multipliers λ 0 with λ i = 1 and 3 0 2 0 0 3 2 0 x 3 x = λ 1 + λ 2 + λ 3 + λ 4 1 2 3 x 2 1 0 0 1 0 0 2 2 x 1 + 2x 2 + 3x 3 = 6 x 1 + x 2 + x 3 + x 4 = 4 x 1, x 2, x 3, x 4 0
Extreme Points and Basic Feasible Solutions in LP Extreme points play a major role in LP - how can we find them? 1 Split A R m n into invertible B R m m and N R m (n m) A = [ 1 2 3 ] 0 1 1 1 1 B = [ ] 1 2 1 1 and N = [ ] 3 0 1 1 The corresponding variables x B and x N are called basic and nonbasic variables, respectively. We can set x N = 0. 2 Write Ax = b as Bx B + Nx N = b and solve for x B = B 1 b x B = [ ] 1 [ ] 1 2 6 1 1 4 [ ] [ ] 1 2 6 = = 1 1 4 [ ] 2 2 If x B 0 the solution is basic feasible, otherwise infeasible. Exercise: Also try B = {1, 3}, {1, 4}, {2, 3}, {2, 4}, and {3, 4}.
Standard and Canonical Forms of Linear Programs Linear programming seeks to minimize or maximize a linear function subject to a linear system of equations or inequalities. Standard form (min. with equalities and nonneg. variables): minimize subject to c T x Ax = b x 0 Canonical form (min. with greater-or-equal inequalities): minimize subject to c T x Ax b The min -form is wlog. Why? (max c T x = min c T x) Exercise: The Fundamental Theorem of LP says if there is an optimal solution, then there is an optimal extreme point. Why?
Optimality Conditions for LPs in Canonical Form Consider the Canonical Form LP: min { c T x : Ax b } Our geometric insight suggests that at an optimal extreme point, the objective normal vector can be written as a linear combination of the normal vectors of the active constraints: c = m y i a i where y i 0 for all i = 1,..., m i=1 To disable inactive constraints we need y i = 0 if a T i x > b: y i (a T i x b i ) = 0 for all i = 1,..., m These are exactly the first-order conditions from calculus for the Lagrangean function L(x, y) = c T x y T (Ax b). Note that normal vectors of hyperplanes {a T i x = b} correspond to the gradient vectors of the linear functions g i (x) = a T i x b i.
LP Algorithms and Computational Complexity Based on the Fundamental Theorem of LP, it suffices to 1 compute the extreme points; 2 evaluate c T x at the basic feasible solutions; 3 select the smallest (largest) such value as the min (max). Quiz: This approach is computationally impractical. Why? Answer: If A R m n, then the number of extreme points is ( n ) m = n! m!(n m)! in standard form - grows exponentially ( n+m ) m = (n+m)! m!n! in canonical form (m slack columns) Later, we will learn two algorithms that are efficient in practice: Dantzig s simplex method: based on fundamental theorem Newer interior-point methods: based on Newton s method applied to the nonlinear system of optimality conditions