Linear Programming MS-E2140 Motivations and background Lecture 1 (course book chapters 1.1-1.4) Linear programming problems and examples Problem manipulations and standard form problems Graphical representation of linear programming problems Modeling absolute values Modeling piecewise linear convex functions v. 2.0
Motivations Linear Programming (LP) problems form an important class of optimization problems with many practical applications. Some example applications: Production planning, resource allocation, investment decisions, military operations, scheduling, transportation, logistics, inventory management, game theory... Efficient LP solution methods are nowadays routinely used within optimization packages to solve even very large problems The Simplex algorithm for solving Linear Programs (Dantzig, 1947) is considered one of the top 10 algorithms developed in the 20-th century (http://www.siam.org/pdf/news/637.pdf) LP theory and duality form the basis to the development of more sophisticated methods for solving hard optimization problems
Historical background A rapid and systematic development of Linear Programming as a practical tool for modeling and solving optimization problems started with the invention of the Simplex algorithm in 1947 The Simplex was invented by Dantzig to solve military planning problems It is one of the most practically effective methods for solving LPs (earlier methods are due to Fourier, 1824, and de la Vallee Poussin, 1910) LP models were also studied in economics in the late 30s by Leonid Kantorovich. Tjalling Koopmans (nobel prize in 1975 with Kantorovich) and Wassily Leontief (nobel prize in 1973) also played an important role The work of von Neumann in game theory (1928) and duality also proved to have strong connections with the fundamental theory of linear programming Paper: Dantzig, G. (2002). Linear Programming, Operations Research, Vol. 50, No.1, pp. 42-47, http://www.jstor.org/stable/3088447
Linear Programming problems (LPs) Minimize or maximize a linear objective function subject to a set of m linear constraints Cost coefficient Minimize z = c 1 x 1 + c 2 x 2 + + c n x n subject to Objective function Can be, or = a 11 x 1 + a 12 x 2 + + a 1n x n b 1 a 21 x 1 + a 22 x 2 + + a 2n x n b 2............ Constraints a m1 x 1 + a m2 x 2 + + a mn x n = b m x 1, x 2, x n 0 Decision variables Non-negativity constraints In general, not all variables are required to be 0, some may be required to be 0 or have no restriction at all
A Linear Programming problem can be expressed in matrix form Minimize z = c 1 x 1 + c 2 x 2 + + c n x n subject to a 11 x 1 + a 12 x 2 + + a 1n x n b 1............ a m1 x 1 + a m2 x 2 + + a mn x n b m x 1, x 2, x n 0 Matrix form Minimize z = c x subject to Ax b x 0 Notation: c = c 1, c 2,, c n R n x = (x 1, x 2,, x n ) R n b = (b 1, b 2,, b m ) R m Constraint matrix A = a 1 a 11 a 1n R m n a m1 a mn We will write a i to denote the i-th row of A so that the i-th constraint can be written as a i x b i Note: The symbol denotes transposition
Some terminology For an LP with a minimization objective function: A vector x that satisfies all the constraints is called feasible solution The set of all the feasible solutions is called feasible region A feasible solution x that minimizes the objective function is called optimal solution Optimal solution: A vector x such that x is feasible and c x c x for any other feasible vector x If x is optimal, the value c x is called optimal cost
Example of LP: Product blending A manufacturer of plastics is planning to blend a new product by mixing four chemical compounds Each compound contains three chemicals A, B, and C in different percentages Table 1 gives for each compound its cost/kg. and the % of each chemical in it Table 1 Comp. 1 Comp. 2 Comp. 3 Comp. 4 % of A 30 10 35 25 % of B 20 65 35 40 Comp. = compound % of C 40 15 25 30 Cost/Kg. 20 30 20 30 The new product must contain 25% of element A, at least 35% of element B, and at least 20% of element C Moreover, to avoid side effects compounds 1 and 2 cannot exceed 25% and 30% of the total, respectively What is the cheapest mix of compounds for blending 1 Kg. of product?
Decision variables x i : fraction of comp. i (i = 1,, 4) used to produce 1 Kg. of product (example: x i = 0.5 means that 1 Kg. of product has 50% of comp. i) Objective function minimize z = 20x 1 + 30x 2 + 20x 3 + 30x 4 Constraints x 1 + x 2 + x 3 + x 4 = 1 Total cost to produce one Kg. of the new product % of comp. 1 + + % of comp. 4 = 100% 30x 1 + 10x 2 + 35x 3 + 25x 4 = 25 20x 1 + 65x 2 + 35x 3 + 40x 4 35 40x 1 + 15x 2 + 25x 3 + 30x 4 20 x 1 0.25, x 2 0.30 x 1, x 2, x 3, x 4 0 exactly 25% of element A at least 35% of element B at least 20% of element C at most 25% of comp. 1 at most 30% of comp. 2 non-negativity
The product blending problem in matrix form: Minimize z = c x x 1 s.t. A 1 x = b 1 A 2 x b 2 A 3 x b 3 1 1 1 1 30 10 35 25 A 1 x 2 x 3 x 4 x = 1 25 b 1 x 0 where 1 1 1 1 A 1 = 30 10 35 25 A 2 = 20 65 35 40 40 15 25 30 1 0 0 0 A 3 = 0 1 0 0 c = 20 30 20 30 b 1 = b 2 = b 3 = 1 25 35 20 0.25 0.30 20 65 35 40 40 15 25 30 A 2 1 0 0 0 0 1 0 0 A 3 x 1 x 2 x 3 x 4 x x 1 x 2 x 3 x 4 x 35 20 b 2 0.25 0.30 b 3
Example of LP: Production planning A food firm is planning the production for the next 4 months A warehouse is available to stock food each month. The maximum storage capacity of the warehouse is 100 tons. Each ton in stock at the end of any month costs 5 EUR On month i, it is possible to produce at most p i tons at cost c i for each ton On month i, an extra production of q i tons is possible at extra cost: Each ton produced in excess of p i in month i has an additional cost of e i The firm has contracted to provide d i tons on each month i The warehouse is empty at the beginning of the first month, and must be empty at the end of the last month The regular (non-extra) production in each month must be at last 10% of the total production of the first three months (balanced production) What is the production/stock policy that minimizes the total cost?
Example data Row i gives the monthly production costs, demand, and capacities of month i Month (i) Production cost (c i ) Decision variables Demand (d i ) x i : Regular production on month i = 1,, 4 s i : Extra production on month i = 1,, 4 y i : Warehouse stock at the end of month i = 1,, 4, (note that y 4 must be fixed to 0) Max. production (p i ) Max. extra production (q i ) Extra production cost (e i ) 1 10 120 140 50 6 2 10 160 150 75 6 3 10 300 140 70 6 4 10 200 160 80 6
Production planning example: Mathematical formulation 4 c i + e i minimize z = (10x i + 16s i + 5y i ) i=1 Total cost Subject to x 1 + s 1 = 120 + y 1 x 2 + s 2 + y 1 = 160 + y 2 x 3 + s 3 + y 2 = 300 + y 3 The production in month i must equal the demand of that month plus the stock at the end of the month x 4 + s 4 + y 3 = 200 x i (s 1 + x 1 + s 2 + x 2 + s 3 + x 3 ) 0.1, x 1 140, x 2 150, x 3 140, x 4 160 s 1 50, s 2 75, s 3 70, s 4 80 i = 1,, 4 Balanced production Maximum regular and extra production x i 0, s i 0, i = 1,, 4, y i 0, i = 1,, 3, y 4 = 0
Problem manipulations Minimize c x is equivalent to Maximize c x An equality constraint a 11 x 1 + + a 1n x n = b 1 is equivalent to: a 11 x 1 + + a 1n x n b 1 and a 11 x 1 + + a 1n x n b 1 An inequality constraint a 11 x 1 + + a 1n x n b 1 is equivalent to: (a 11 x 1 + + a 1n x n ) b 1 An inequality constraint a 11 x 1 + + a 1n x n b 1 is equivalent to: a 11 x 1 + + a 1n x n + s 1 = b 1 where s 1 0 is a new variable called slack variable Similarly, a 11 x 1 + + a 1n x n b 1 is equivalent to: a 11 x 1 + + a 1n x n s 1 = b 1, where s 1 0 is a slack variable
A free variable x i (free = not restricted to be 0 or 0) can be replaced with x + i x i where x + i 0 and x i 0 are non-negative variables Examples of problem manipulations 3x 1 + 2x 2 6 2x 1 + 9x 2 8 free minimize z = 2x 1 + 5x 2 minimize z = 2x 1 + 5(x 2 + x 2 ) Subject to Subject to 3x 1 + 2(x + 2 x 2 ) 6 2x 1 + 9(x + 2 x 2 ) 8 x 1 0 x 1, x 2 +, x 2 0 minimize z = 2x 1 x 2 + 4x 3 Subject to x 1 + x 2 + x 4 2 3x 2 x 3 = 5 x 3 + x 4 3 x 1 0, x 3 0 minimize z = 2x 1 x 2 + 4x 3 Subject to x 1 x 2 x 4 2 3x 2 x 3 5 3x 2 + x 3 5 x 3 +x 4 3 x 1 0, x 3 0
Standard form problems By using the previous transformations we can always express any LP in the following form, called standard form All variables are restricted to take non-negative values Minimize z = c x Subject to Ax = b x 0 All constraints are equality constraints Starting from any LP, we can rewrite it in standard form as follows: 1. Replace each free variable x i with (x i + x i ) where x i + 0, x i 0 2. Transform any inequality constraint into an equality constraint by adding slack variables Any LP and its standard form are equivalent: Given a feasible solution to the original LP we can construct a feasible solution to its standard form with same cost, and vice versa
An LP (not in standard form) minimize z = 2x 1 + 4x 2 subject to x 1 + x 2 3 3x 1 + 2x 2 = 14 x 1 0 The same LP in standard form minimize z = 2x 1 + 4x + 2 4x 2 subject to x 1 + x + 2 x 2 s 1 = 3 3x 1 + 2x + 2 2x 2 = 14 x 1, x + 2, x 2, s 1 0 The Simplex algorithm is designed to solve LPs in standard form This is because it works by iteratively applying the following operations: 1. multiply the coefficients and right hand side of a constraint by a nonzero real number 2. apply operation 1 to a constraint and sum the result to another one The operations 1 and 2 when applied to a system of linear equations leave unchanged the set of solutions
Graphical representation of an LP By transforming equality constraints into inequalities, we can also rewrite any LP in the following general form Minimize z = c x subject to Ax b where eventual non-negativity constraints are included in the definition of A. For example: minimize z = 3x 1 2x 2 + x 3 subject to 2x 1 x 2 1 x 2 + x 3 5 x 1 0 x 3 0 has A = 2 1 0 0 1 1 1 0 0 0 0 1 x = (x 1 x 2 x 3 ) b = (1 5 0 0) Each constraint of an LP in this form with n variables defines a region of R n, called halfspace, containing all x R n satisfying the constraint The feasible region is the intersection of all these halfspaces
For an LP in general form with two or three variables, we can visualize its feasible region, and even solve the problem graphically: minimize z = x 1 3x 2 subject to x 1 x 2 6 (1) x 1 2x 2 8 (2) x 1 + x 2 2 (3) x 1 0 4 x 2 0 (5) Halfspace defined by x 1 + x 2 2 x 2 Halfspace defined by x 1 + x 2 2 x 1 + x 2 = 2 x 1 The feasible region is the intersection of the halfspaces defined by (1) (5) (3) (5) x 2 (2) feasible region (1) (4) For any value z, all the solutions x with cost c x = z lie on the line c x = x 1 3x 2 = z called iso-profit line This iso-profit line is perpendicular to the cost vector c c x 1
x 2 Graphical solution of an LP (n = 2) Optimal solution x = ( 4 3 14 3 ) of cost z = 38/3 c = ( 1 3) x 1 3x 2 = 38/3 (z = 38/3) x 1 3x 2 = 12 (z = 12) x 1 3x 2 = 6 (z = 6) Isoprofit line for z = 6 x 1 3x 2 = 0 (z = 0) Isoprofit line for z = 0 We start with z = 0 and draw the iso-profit line for z = 0 The value of z decreases in the direction of c. To minimize z we move the line parallel to itself in that direction The minimum value of z is obtained when the line cannot be moved further without leaving the feasible region When this happens the line intersects a corner point of the feasible region which is an optimal solution
It is not always the case that an LP has a unique optimal solution Consider the following LP: x 2 x 1 + x 2 = 1 minimize z = c 1 x 1 + c 2 x 2 subject to x 1 + x 2 1 x 1 0 x 2 0 Feasible region x 1 Depending on the choice of c, the following cases can occur: 1) Unique optimal solution 2) Bounded set of optimal solutions x 2 If c = (1 1) there is a unique optimal solution x x 2 If c = (1 0) any x = (0 x 2 ) with 0 x 2 1 is optimal (0 1) c = (1 1) x = (0 x 2 ) with 0 x 2 1 x = (0 0) x 1 (0 0) c = (1 0) x 1 c x = x 1 + x 2 = 0 (z = 0)
3) Unbounded set of optimal solutions 4) No optimal solution: Optimal cost x 2 If c = (0 1) any x = (x 1 0) with x 1 0 is optimal x 2 If c = ( 1 1) there is no optimal solution: z for x 1, x 2 the optimal cost is c = (0 1) x 1 x 1 x = (x 1 0) with x 1 0 c = ( 1 1) Finally, the feasible region can also be empty, in which case there are no feasible solutions and the LP is said to be infeasible: Example of infeasible LP: x 2 minimize z = c 1 x 1 + c 2 x 2 subject to x 1 + x 2 1 x 1 x 2 1 x 1 0 x 2 0 x 1 x 2 1 x 1 x 1 + x 2 1
Modeling absolute values LPs can be used to model situations where variables represent absolute values under some assumptions on the sign of their cost coefficients Problem P minimize σ n i=1 c i x i subject to Ax b where x = (x 1,, x n ) and c i 0, i = 1,, n This is not an LP Observation: x i is the smallest number y i satisfying y i x i and y i x i Problem P minimize σ n i=1 c i y i subject to Ax b y i x i, y i x i, i = 1,, n This is an LP Solving P (which is an LP) is equivalent to solving P This is because c i 0 and we are minimizing: If a solution has y i > x i and y i > x i then we can reduce y i and get a feasible solution with lower cost Thus, any optimal solution of P must have y i = max x i, x i = x i
Another possibility to solve P in the previous slide as an LP is to: replace x i with (x i + x i ) where x i + 0, x i 0 replace x i with (x i + + x i ) In this way we obtain the LP: Example: if x i = 4 we can write x i + = 0 and x i = 4 so that x i = x i + x i and x i = x i + + x i minimize σ n i=1 c i x i + + x i s.t. Ax + Ax b x +, x 0 where x + = (x + 1,, x + n ) and x = (x 1,, x n ) Since we are minimizing and c i 0, an optimal solution must have x + i = 0 or x i = 0, i, because otherwise we could reduce both x + i and x i by the same amount and obtain a better feasible solution
Example. Consider the problem: Problem P Minimize z = 2 x 1 + x 2 subject to x 1 + x 2 4 Using the reformulations of the previous slides we obtain: Minimize z = 2y 1 + x 2 subject to x 1 + x 2 4 y 1 x 1 y 1 x 1 or Minimize z = 2x + 1 + 2x 1 + x 2 subject to x + 1 x 1 + x 2 4 x + 1 0 x 1 0
Modeling piecewise linear functions A function f: R n R is called convex if for every x, y R n, and for every λ [0,1], we have f λx + 1 λ y λf x + (1 λ)f y f is concave if it satisfies the above with " " replaced by " Note: All points λx + 1 λ y lie on the line segment joining x and y; Informally, f is convex if its graph lies below the line connecting x and y f x λf x + (1 λ)f y, for λ [0,1] f x f y f y x y x y A convex function f λx + (1 λ)y, for λ [0,1] A concave function
A function of the form f x = max (c i x + d i ) is called piecewise i=1,,m linear convex (or piecewise linear concave if max is replaced by min). Example with m = 3, and x R 1 f x = max 1 x, 2 x + 2, x + 5 is piecewise linear convex 2 5 2 Piecewise linear convex (or concave) functions can be used to approximate convex (or concave) functions f x f x f x = x + 5 2 f x = 1 2 x 1 f x = 2 5 x + 2 A piecewise linear convex function x A convex function approximated by a piecewise linear convex function x
Consider the following problems (which are not LPs): The minimization of a piecewise linear convex function: (1) Minimize z = subject to max (c i x + d i ) i=1,,m Ax b The maximization of a piecewise linear concave function: (2) Maximize z = subject to min (c i x + d i ) i=1,,m Ax b Problems (1) and (2) can be transformed into LPs The idea is to introduce z explicitly as a variable, and then rewrite (1) and (2) as the following LPs: (1) Minimize z (2) Maximize z subject to Ax b z c i x + d i, i = 1,, m subject to Ax b z c i x + d i, i = 1,, m
Modeling with piecewise linear functions: Example Consider a firm producing and selling a product. A typical strategy is to introduce economies of scale to incentivate bulk buys Suppose the unit selling price is set to 10 EUR for the first 1000 units, but is reduced to 7 EUR for any amount in excess to that Consider variables x i modeling the amount sold in month i = 1,, n What is the profit p i (x i ) made for selling x i in month i? p i (x i ) = 10x i 10 1000 + 7(x i 1000) if 0 x i 1000 if x i 1000 3000 + 7x i p i (x i ) is a piecewise linear concave function since is can be written as p i x i = min 10x i, 7x i + 3000
Let x = (x 1,, x n ) and suppose there are constraints Ax b. We wish to maximize the profit over the n months. The problem is: n Maximize p i (x i ) i=1 subject to Ax b x 0 which can be transformed into the following LP n Maximize i=1 p i subject to Ax b p i is a continuous variable which models p i (x i ) p i 10x i, p i 7x i + 3000 i = 1,, n x 0, p i 0, i