Linear Programming and the Simplex method Harald Enzinger, Michael Rath Signal Processing and Speech Communication Laboratory Jan 9, 2012 Harald Enzinger, Michael Rath Jan 9, 2012 page 1/37
Outline Introduction to Linear Programming Simplex Basics Computational Details Demonstration Dual problem of LP Harald Enzinger, Michael Rath Jan 9, 2012 page 2/37
Outline Introduction to Linear Programming Simplex Basics Computational Details Demonstration Dual problem of LP Harald Enzinger, Michael Rath Jan 9, 2012 page 3/37
Formulation of a Linear Program minimize subject to c T x Ax b Cx = d The objective function c T x is a linear function of n decision variables x 1 to x n There can be linear inequality and equality constraints The constraints define a feasible set of solutions The goal is to find a feasible solution that minimizes the objective function Harald Enzinger, Michael Rath Jan 9, 2012 page 4/37
Geometric Interpretation of a Linear Equation in 2D ( ) a T a1 x = b with a = a 2 x = ( x1 x 2 a 1 x 1 + a 2 x 2 = b x 2 = a 1 a 2 x 1 + b a 2 ) Harald Enzinger, Michael Rath Jan 9, 2012 page 5/37
Proof of Geometric Interpretation a T x = a T (x + x ) = a T x + a T x = a x cos(0 ) + a x cos(90 ) = a x = b x = b a Harald Enzinger, Michael Rath Jan 9, 2012 page 6/37
Generalization to more Dimensions a T x = b dim x = n n = 3: Plane n > 3: Hyperplane a T x b or a T x b Halfspace that is defined by (Hyper-) Plane Harald Enzinger, Michael Rath Jan 9, 2012 page 7/37
Set of Equations a 11 a 1n x 1... a m1 a mn x n = b 1. b m Ax = b solution is an affine subspace dimension of solution space: n rank(a) Ax b Intersection of m Halfspaces Harald Enzinger, Michael Rath Jan 9, 2012 page 8/37
Visualization of a Linear Program Polyhedron represents feasible region (Hyper-) planes represent constant objective function value Objective function value is proportional to distance from origin Optimal solution lies on the surface of the polyhedron Optimal solution lies in a corner of the polyhedron Optimal solution is a global optimum Harald Enzinger, Michael Rath Jan 9, 2012 page 9/37
Special Cases of feasible region No intersection of halfspaces Optimal solution is unbounded Harald Enzinger, Michael Rath Jan 9, 2012 page 10/37
Canonical Form of a Linear Program maximize subject to c T x Ax b x 0 minimization of c T x is equal to maximization of c T x constraint a T x b is equal to constraint a T x b constraint a T x = b is equal to constraints a T x b and a T x b unbounded variable x i can be split into two bounded variables: x i unbounded x i = x i1 x i2 x i1 0 x i2 0 Harald Enzinger, Michael Rath Jan 9, 2012 page 11/37
Transformation to Standard Form Transform inequations to equations by introducing slack variables x n+1 to x n+m a T i x b i a T i x + x n+i = b i Ax b (A I)x = b a 11 a 1n 1 0... 1. a m1 a mn 0 1 x 1. x n x n+1. x n+m = b 1. b m Harald Enzinger, Michael Rath Jan 9, 2012 page 12/37
Basic Solutions a 11 a 1n 1 0 b 1... 1.. a m1 a mn 0 1 b m A basis is a subset of m linearly independent columns Basic variables x B are variables that belong to the basis Non-basic variables x N are the remaining variables A basic solution is found by setting x B = A 1 B b and x N = 0 e.g. x T B = (x n+1 x n+m ) x T N = (x 1 x n ) x B = b x N = 0 Harald Enzinger, Michael Rath Jan 9, 2012 page 13/37
Corners of Polyhedron Relation of Basic Solutions and Corners Every basic solution corresponds to a corner of the polyhedron. x N = 0 Solution lies in intersection of hyperplanes H j, j N x B = A 1 B b is unique Solution is unique A unique intersection of n hyperplanes must be a corner. Additional Properties A basic solution / corner is feasible if all x B 0 A basic solution / corner is degenerated if there is an x B = 0 Harald Enzinger, Michael Rath Jan 9, 2012 page 14/37
Example for Basic Solutions and Edges n = 2 variables m = 3 constraints x 1 + x 2 4 2x 1 x 2 3 x 2 1 x 1 0 x 2 0 Harald Enzinger, Michael Rath Jan 9, 2012 page 15/37
Example for Basic Solutions and Edges 1 1 1 0 0 4 2 1 0 1 0 3 0 1 0 0 1 1 N = {1, 2} B = {3, 4, 5} Edge is feasible and not degenerated 1 0 0 x 3 4 0 1 0 x 4 = 3 x T = (0, 0, 4, 3, 1) 0 0 1 x 5 1 Harald Enzinger, Michael Rath Jan 9, 2012 page 16/37
Example for Basic Solutions and Edges 1 1 1 0 0 4 2 1 0 1 0 3 0 1 0 0 1 1 N = {1, 3} B = {2, 4, 5} Edge is not feasible and not degenerated 1 0 0 x 2 4 1 1 0 x 4 = 3 x T = (0, 4, 0, 7, 3) 1 0 1 x 5 1 Harald Enzinger, Michael Rath Jan 9, 2012 page 17/37
Example for Basic Solutions and Edges 1 1 1 0 0 4 2 1 0 1 0 3 0 1 0 0 1 1 N = {4, 5} B = {1, 2, 3} Edge is feasible and degenerated 1 1 1 x 1 4 2 1 0 x 2 = 3 x T = (2, 1, 0, 0, 0) 0 1 0 x 3 1 Harald Enzinger, Michael Rath Jan 9, 2012 page 18/37
Outline Introduction to Linear Programming Simplex Basics Computational Details Demonstration Dual problem of LP Harald Enzinger, Michael Rath Jan 9, 2012 page 19/37
The Simplex method Basic Idea Start at corner point initial basic solution Move along edge increase one variable at a time Select variable with largest improvement of z entering variable Move to next feasible corner point select leaving variable Repeat until optimal corner point reached no more improvement of z Harald Enzinger, Michael Rath Jan 9, 2012 page 20/37
The Simplex method Basic Idea Harald Enzinger, Michael Rath Jan 9, 2012 page 21/37
The Simplex method Requirements LP in standard form: maximize subject to Convert LP to standard form a i x b i: z = c x Ax = b x 0, b 0 Introduce slack variable a i x + s i = b i Example: 6x 1 + 4x 2 24 6x 1 + 4x 2 + s 1 = 24 a i x b i: Introduce surplus variable a i x S i = b i Example: x 1 + x 2 800 x 1 + x 2 S 1 = 800 Harald Enzinger, Michael Rath Jan 9, 2012 page 22/37
The Simplex method - Computational Details Initialization Build tableau for canonical form Use slack variables as starting basic solution Basic x 1... x n s 1... s m Solution z c 1... c n 0... 0 0 z-row s 1 a 11... a 1n 1... 0 b 1 s 1 -row............. s m a m1... a mn 0... 1 b m s m -row z-row corresponds to z c 1 x 1 c 2 x 2... c n x n = 0 Harald Enzinger, Michael Rath Jan 9, 2012 page 23/37
The Simplex method - Computational Details Optimality condition Choose variable to enter the basic solution Take the one with the most negative coefficient in objective equation (z-row) If there is none with negative coefficient, optimality has been reached Basic x 1 x 2 s 1 s 2 s 3 s 4 Solution z 5 4 0 0 0 0 0 z-row s 1 6 4 1 0 0 0 24 s 1 -row s 2 1 2 0 1 0 0 6 s 2 -row s 3 1 1 0 0 1 0 1 s 3 -row s 4 0 1 0 0 0 1 2 s 4 -row Harald Enzinger, Michael Rath Jan 9, 2012 page 24/37
The Simplex method - Computational Details Feasibility condition Choose variable to leave the basic solution Take the one with the minimum non-negative ratio Ratios of {solution/entering variable coefficient} correspond to intercerpts of constraints with entering variable Basic x 1 x 2 s 1 s 2 s 3 s 4 Solution Ratio z 5 4 0 0 0 0 0 24 s 1 6 4 1 0 0 0 24 6 = 4 6 s 2 1 2 0 1 0 0 6 1 = 6 s 3 1 1 0 0 1 0 1 < 0 s 4 0 1 0 0 0 1 2 Harald Enzinger, Michael Rath Jan 9, 2012 page 25/37
The Simplex method - Computational Details Swapping Entering and Leaving Variable Replace leaving var. in basic solution with entering var. Basic x 1 x 2 s 1 s 2 s 3 s 4 Solution z 5 4 0 0 0 0 0 s 1 6 4 1 0 0 0 24 s 1 -row s 2 1 2 0 1 0 0 6 s 2 -row s 3 1 1 0 0 1 0 1 s 3 -row s 4 0 1 0 0 0 1 2 s 4 -row Harald Enzinger, Michael Rath Jan 9, 2012 page 26/37
The Simplex method - Computational Details Swapping Entering and Leaving Variable New pivot row = Current pivot row / Pivot element Basic x 1 x 2 s 1 s 2 s 3 s 4 Solution z 5 4 0 0 0 0 0 x 1 6 4 1 0 0 0 24 x 1 -row s 2 1 2 0 1 0 0 6 s 2 -row s 3 1 1 0 0 1 0 1 s 3 -row s 4 0 1 0 0 0 1 2 s 4 -row Harald Enzinger, Michael Rath Jan 9, 2012 page 26/37
The Simplex method - Computational Details Swapping Entering and Leaving Variable New row = (Current row) - (its pivot col.coeff.) (New pivot row) Basic x 1 x 2 s 1 s 2 s 3 s 4 Solution z 5 4 0 0 0 0 0 x 1 1 2 3 1 6 0 0 0 4 x 1 -row s 2 1 2 0 1 0 0 6 s 2 -row s 3 1 1 0 0 1 0 1 s 3 -row s 4 0 1 0 0 0 1 2 s 4 -row Harald Enzinger, Michael Rath Jan 9, 2012 page 26/37
The Simplex method - Computational Details Swapping Entering and Leaving Variable Solution of iteration Basic x 1 x 2 s 1 s 2 s 3 s 4 Solution 5 z 0 2 3 6 0 0 0 20 2 1 x 1 1 3 6 0 0 0 4 x 1 -row 4 s 2 0 3 1 6 1 0 0 2 s 2 -row 5 1 s 3 0 3 6 0 1 0 5 s 3 -row s 4 0 1 0 0 0 1 2 s 4 -row Harald Enzinger, Michael Rath Jan 9, 2012 page 26/37
The Simplex method - Computational Details Getting initial basic feasible solution (BFS) For canonical form one can take slack variables for initial BFS (=) Constraints a i x = b i: Introduce artificial variable a i x + R i = b i ( ) Constraints a i x b i: Introduce surplus and artificial variable a i x S i + R i = b i Dealing with artificial variables Eliminate artificial variables using standard simplex to get BFS M-method or Two-phase Method Harald Enzinger, Michael Rath Jan 9, 2012 page 27/37
The Simplex method - Computational Details M-method Introduce high penalty into objective function for artificial variables Maximize z = c x MR Choose M accordingly to guarantee drop out of artificial variables Large M can result in roundoff errors that impair accuracy Basic x 1 x 2 x 3 R 1 R 2 Solution z 4 1 0 100 100 0 R 1 3 1 0 1 0 3 R 2 4 3 1 0 1 6 Normalization Basic x 1 x 2 x 3 R 1 R 2 Solution z 696 399 100 0 0 900....... Harald Enzinger, Michael Rath Jan 9, 2012 page 28/37
The Simplex method - Computational Details Two-phase Method Solve the LP in two phases: Phase 1: Introduce new objective function to minimize the sum of artificial variables {Minimize r = i R i} {Maximize r = i R i} Phase 2: Perform usual simplex with solution obtained from Phase 1 Basic x 1 x 2 x 3 R 1 R 2 Solution r 0 0 0 1 1 0 R 1 3 1 0 1 0 3 R 2 4 3 1 0 1 6 Normalization Basic x 1 x 2 x 3 R 1 R 2 Solution r 7 4 1 0 0 9....... Harald Enzinger, Michael Rath Jan 9, 2012 page 29/37
The Simplex method - Special Cases Degeneracy Happens if tie occurs for minimum ratio in Feasibility condition At least one basic variable will be zero in next iteration Model has at least one redundant constraint (overdetermined point) Cycling if objective value doesnt improve Alternative Optima Objective function parallel to constraint All points between corner points optimal solutions Harald Enzinger, Michael Rath Jan 9, 2012 page 30/37
The Simplex method - Demonstration Solving toy example graphically minimize x 1 x 2 subject to x 1 x 2 2 x 1 x 2 5 3x 1 x 2 18 3x 1 + x 2 27 x 1 + 5x 2 25 x 1 + x 2 3 4x 1 + x 2 0 Harald Enzinger, Michael Rath Jan 9, 2012 page 31/37
Harald Enzinger, Michael Rath Jan 9, 2012 page 32/37
Harald Enzinger, Michael Rath Jan 9, 2012 page 32/37
Harald Enzinger, Michael Rath Jan 9, 2012 page 32/37
Harald Enzinger, Michael Rath Jan 9, 2012 page 32/37
Harald Enzinger, Michael Rath Jan 9, 2012 page 32/37
Harald Enzinger, Michael Rath Jan 9, 2012 page 32/37
Harald Enzinger, Michael Rath Jan 9, 2012 page 32/37
Harald Enzinger, Michael Rath Jan 9, 2012 page 32/37
Harald Enzinger, Michael Rath Jan 9, 2012 page 32/37
Harald Enzinger, Michael Rath Jan 9, 2012 page 32/37
Outline Introduction to Linear Programming Simplex Basics Computational Details Demonstration Dual problem of LP Harald Enzinger, Michael Rath Jan 9, 2012 page 33/37
Dual of LP Definition Dual of LP defined from primal (original) LP model Optimal solution of one problem also provides solution to the other Rules to construct dual problem Define dual variable for each primal constraint Define dual constraint for each primal variable Primal constraint coefficients define left-hand side coefficients of dual constraint and its objective coefficient defines the right-hand side Objective coefficients of dual equal right-hand side of primal constraint equations Type of optimization switches (max min) Dual constraint type is determined by primal optimization type (min, max ) Harald Enzinger, Michael Rath Jan 9, 2012 page 34/37
Dual of LP Example Use rules on primal in equation form (Standard form) Primal in equation form Dual variables Minimize z = 15x 1 + 12x 2 + 0x 3 + 0x 4 subject to x 1 + 2x 2 x 3 + 0x 4 = 3 y 1 2x 1 4x 2 + 0x 3 + x 4 = 5 y 2 x 1, x 2, x 3, x 4 0 Dual Problem subject to Maximize w = 3y 1 + 5y 2 y 1 +2y 2 15 2y 1 4y 2 12 y 1 0 y 2 0 Harald Enzinger, Michael Rath Jan 9, 2012 page 35/37
References Rainer Burkard, Lecture Notes: Mathematische Optimierung, http://www.opt.math.tu-graz.ac.at/ hatzl/vorlesung/mathoptss11/opt.pdf Juncheng Wei, Lecture Notes: Linear Programming, http://www.math.cuhk.edu.hk/ wei/lp11.html H.A. Taha, Operations Research: An Introduction, Pearson Prentice Hall, 8th Edition, 2007 Harald Enzinger, Michael Rath Jan 9, 2012 page 36/37