Non-Lecture I: Linear Programming. Th extremes of glory and of shame, Like east and west, become the same.

Transcription

1 The greatest flood has the soonest ebb; the sorest tempest the most sdden calm; the hottest love the coldest end; and from the deepest desire oftentimes enses the deadliest hate. Th extremes of glory and of shame, Like east and est, become the same. Socrates Samel Btler, Hdibras Part II, Canto I (c. 1670) Extremes meet, and there is no better example than the haghtiness of hmility. Ralph Waldo Emerson, Greatness, in Letters and Social Aims (1876) I Linear Programming The maximm flo/minimm ct problem is a special case of a very general class of problems called linear programming. Many other optimization problems fall into this class, inclding minimm spanning trees and shortest paths, as ell as several common problems in schedling, logistics, and economics. Linear programming as sed implicitly by Forier in the early 1800s, bt it as first formalized and applied to problems in economics in the 1930s by Leonid Kantorovich. Kantorivich s ork as hidden behind the Iron Crtain (here it as largely ignored) and therefore nknon in the West. Linear programming as rediscovered and applied to shipping problems in the early 1940s by Tjalling Koopmans. The first complete algorithm to solve linear programming problems, called the simplex method, as pblished by George Dantzig in Koopmans first proposed the name linear programming" in a discssion ith Dantzig in Kantorovich and Koopmans shared the 1975 Nobel Prize in Economics for their contribtions to the theory of optimm allocation of resorces. Dantzig did not; his ork as apparently too pre. Koopmans rote to Kantorovich sggesting that they refse the prize in protest of Dantzig s exclsion, bt Kantorovich sa the prize as a vindication of his se of mathematics in economics, hich his Soviet colleages had ritten off as a means for apologists of capitalism. A linear programming problem asks for a vector x IR d that maximizes (or eqivalently, minimizes) a given linear fnction, among all vectors x that satisfy a given set of linear ineqalities. The general form of a linear programming problem is the folloing: maximize c j x j sbject to a i j x j b i a i j x j = b i a i j x j b i for each i = 1.. p for each i = p p + q for each i = p + q n Here, the inpt consists of a matrix A = (a i j ) IR n d, a colmn vector b IR n, and a ro vector c IR d. Each coordinate of the vector x is called a variable. Each of the linear ineqalities is called a constraint. The fnction x c x is called the objective fnction. I ill alays se d to denote the nmber of variables, also knon as the dimension of the problem. The nmber of constraints is sally denoted n. 1

2 A linear programming problem is said to be in canonical form 1 if it has the folloing strctre: maximize c j x j sbject to a i j x j b i x j 0 for each i = 1.. n for each j = 1.. d We can express this canonical form more compactly as follos. For to vectors x = (x 1, x 2,..., x d ) and y = (y 1, y 2,..., y d ), the expression x y means that x i y i for every index i. s.t. Ax b x 0 Any linear programming problem can be converted into canonical form as follos: For each variable x j, add the eqality constraint x j = x + j x j and the ineqalities x + j 0 and x j 0. Replace any eqality constraint j a i j x j = b i ith to ineqality constraints j a i j x j b i and j a i j x j b i. Replace any pper bond j a i j x j b i ith the eqivalent loer bond j a i j x j b i. This conversion potentially doble the nmber of variables and the nmber of constraints; fortnately, it is rarely necessary in practice. Another sefl format for linear programming problems is slack form 2, in hich every ineqality is of the form x j 0: s.t. Ax = b x 0 It s fairly easy to convert any linear programming problem into slack form. Slack form is especially sefl in execting the simplex algorithm (hich e ll see in the next lectre). I.1 The Geometry of Linear Programming A point x IR d is feasible ith respect to some linear programming problem if it satisfies all the linear constraints. The set of all feasible points is called the feasible region for that linear program. The feasible region has a particlarly nice geometric strctre that lends some sefl intition to the linear programming algorithms e ll see later. Any linear eqation in d variables defines a hyperplane in IR d ; think of a line hen d = 2, or a plane hen d = 3. This hyperplane divides IR d into to halfspaces; each halfspace is the set of points that satisfy some linear ineqality. Ths, the set of feasible points is the intersection of several hyperplanes 1 Confsingly, some athors call this standard form. 2 Confsingly, some athors call this standard form. 2

3 (one for each eqality constraint) and halfspaces (one for each ineqality constraint). The intersection of a finite nmber of hyperplanes and halfspaces is called a polyhedron. It s not hard to verify that any halfspace, and therefore any polyhedron, is convex if a polyhedron contains to points x and y, then it contains the entire line segment x y. A to-dimensional polyhedron (hite) defined by 10 linear ineqalities. By rotating IR d (or choosing a coordinate frame) so that the objective fnction points donard, e can express any linear programming problem in the folloing geometric form: Find the loest point in a given polyhedron. With this geometry in hand, e can easily pictre to pathological cases here a given linear programming problem has no soltion. The first possibility is that there are no feasible points; in this case the problem is called infeasible. For example, the folloing LP problem is infeasible: maximize x y sbject to 2x + y 1 x + y 2 x, y 0 An infeasible linear programming problem; arros indicate the constraints. The second possibility is that there are feasible points at hich the objective fnction is arbitrarily large; in this case, e call the problem nbonded. The same polyhedron cold be nbonded for some objective fnctions bt not others, or it cold be nbonded for every objective fnction. A to-dimensional polyhedron (hite) that is nbonded donard bt bonded pard. 3

4 I.2 Example 1: Shortest Paths We can compte the length of the shortest path from s to t in a eighted directed graph by solving the folloing very simple linear programming problem. maximize d t sbject to d s = 0 d v d l v Here, l v is the length of the edge v. Each variable d v represents a tentative shortest-path distance from s to v. The constraints mirror the reqirement that every edge in the graph mst be relaxed. These relaxation constraints imply that in any feasible soltion, d v is at most the shortest path distance from s to v. Ths, somehat conterintitively, e are correctly maximizing the objective fnction to compte the shortest path! In the optimal soltion, the objective fnction d t is the actal shortest-path distance from s to t, bt for any vertex v that is not on the shortest path from s to t, d v may be an nderestimate of the tre distance from s to v. Hoever, e can obtain the tre distances from s to every other vertex by modifying the objective fnction: maximize v d v sbject to d s = 0 d v d l v There is another formlation of shortest paths as an LP minimization problem sing an indicator variable x v for each edge v. minimize l v x v v sbject to x s x s = 1 x t x t = 1 x v x v = 0 x v 0 for every vertex v s, t Intitively, x v = 1 means v lies on the shortest path from s to t, and x v = 0 means v does not lie on this shortest path. The constraints merely state that the path shold start at s, end at t, and either pass throgh or avoid every other vertex v. Any path from s to t in particlar, the shortest path clearly implies a feasible point for this linear program. Hoever, there are other feasible soltions, possibly even optimal soltions, ith non-integral vales that do not represent paths. Nevertheless, there is alays an optimal soltion in hich every x e is either 0 or 1 and the edges e ith x e = 1 comprise the shortest path. (This fact is by no means obvios, bt a proof is beyond the scope of these notes.) Moreover, in any optimal soltion, even if not every x e is an integer, the objective fnction gives the shortest path distance! I.3 Example 2: Maximm Flos and Minimm Cts Recall that the inpt to the maximm (s, t)-flo problem consists of a eighted directed graph G = (V, E), to special vertices s and t, and a fnction assigning a non-negative capacity c e to each edge e. Or task 4

5 is to choose the flo f e across each edge e, as follos: maximize f s sbject to f s f v f v = 0 f v c v f v 0 for every vertex v s, t Similarly, the minimm ct problem can be formlated sing indicator variables similarly to the shortest path problem. We have a variable S v for each vertex v, indicating hether v S or v T, and a variable X v for each edge v, indicating hether S and v T, here (S, T) is some (s, t)-ct. 3 minimize c v X v v sbject to X v + S v S 0 X v 0 S s = 1 S t = 0 Like the minimization LP for shortest paths, there can be optimal soltions that assign fractional vales to the variables. Nevertheless, the minimm vale for the objective fnction is the cost of the minimm ct, and there is an optimal soltion for hich every variable is either 0 or 1, representing an actal minimm ct. No, this is not obvios; in particlar, my claim is not a proof! I.4 Linear Programming Dality Each of these pairs of linear programming problems is related by a transformation called dality. For any linear programming problem, there is a corresponding dal linear program that can be obtained by a mechanical translation, essentially by sapping the constraints and the variables. The translation is simplest hen the LP is in canonical form: Primal (Π) s.t. Ax b x 0 Dal ( ) min y b s.t. ya c y 0 We can also rite the dal linear program in exactly the same canonical form as the primal, by sapping the coefficient vector c and the objective vector b, negating both vectors, and replacing the constraint matrix A ith its negative transpose. 4 Primal (Π) s.t. Ax b x 0 Dal ( ) max b y s.t. A y c y 0 3 These to linear programs are not qite syntactic dals; I ve added to redndant variables S s and S t to the min-ct program to increase readability. 4 For the notational prists: In these formlations, x and b are colmn vectors, and y and c are ro vectors. This is a somehat nonstandard choice. Yes, that means the dot in c x is redndant. Se me. 5

6 Written in this form, it shold be immediately clear that dality is an involtion: The dal of the dal linear program is identical to the primal linear program Π. The choice of hich LP to call the primal and hich to call the dal is totally arbitrary. 5 The Fndamental Theorem of Linear Programming. A linear program Π has an optimal soltion x if and only if the dal linear program has an optimal soltion y sch that c x = y Ax = y b. The eak form of this theorem is trivial to prove. Weak Dality Theorem. If x is a feasible soltion for a canonical linear program Π and y is a feasible soltion for its dal, then c x yax y b. Proof: Becase x is feasible for Π, e have Ax b. Since y is positive, e can mltiply both sides of the ineqality to obtain yax y b. Conversely, y is feasible for and x is positive, so yax c x. It immediately follos that if c x = y b, then x and y are optimal soltions to their respective linear programs. This is in fact a fairly common ay to prove that e have the optimal vale for a linear program. I.5 Dality Example Before I prove the stronger dality theorem, let me first provide some intition abot here this dality thing comes from in the first place. 6 Consider the folloing linear programming problem: maximize 4x 1 + x 2 + 3x 3 sbject to x 1 + 4x 2 2 3x 1 x 2 + x 3 4 x 1, x 2, x 3 0 Let σ denote the optimm objective vale for this LP. The feasible soltion x = (1, 0, 0) gives s a loer bond σ 4. A different feasible soltion x = (0, 0, 3) gives s a better loer bond σ 9. We cold play this game all day, finding different feasible soltions and getting ever larger loer bonds. Ho do e kno hen e re done? Is there a ay to prove an pper bond on σ? In fact, there is. Let s mltiply each of the constraints in or LP by a ne non-negative scalar vale y i : maximize 4x 1 + x 2 + 3x 3 sbject to y 1 ( x 1 + 4x 2 ) 2 y 1 y 2 (3x 1 x 2 + x 3 ) 4 y 2 x 1, x 2, x 3 0 Becase each y i is non-negative, e do not reverse any of the ineqalities. Any feasible soltion (x 1, x 2, x 3 ) mst satisfy both of these ineqalities, so it mst also satisfy their sm: (y y 2 )x 1 + (4 y 1 y 2 )x 2 + y 2 x 3 2 y y 2. 5 For historical reasons, maximization LPs tend to be called primal and minimization LPs tend to be called dal. This is a pointless religios tradition, nothing more. Dality is a relationship beteen LP problems, not a type of LP problem. 6 This example is taken from Robert Vanderbei s excellent textbook Linear Programming: Fondations and Extensions [Springer, 2001], bt the idea appears earlier in Jens Clasen s 1997 paper Teaching Dality in Linear Programming: The Mltiplier Approach. 6

7 No sppose that each y i is larger than the ith coefficient of the objective fnction: y y 2 4, 4y 1 y 2 1, y 2 3. This assmption lets s derive an pper bond on the objective vale of any feasible soltion: 4x 1 + x 2 + 3x 3 (y y 2 )x 1 + (4 y 1 y 2 )x 2 + y 2 x 3 2 y y 2. ( ) In particlar, by plgging in the optimal soltion (x 1, x 2, x 3 ) for the original LP, e obtain the folloing pper bond on σ : σ = 4x 1 + x 2 + 3x 3 2 y y 2. No it s natral to ask ho tight e can make this pper bond. Ho small can e make the expression 2y y 2 ithot violating any of the ineqalities e sed to prove the pper bond? This is jst another linear programming problem. minimize 2y y 2 sbject to y y y 1 y 2 1 y 2 3 y 1, y 2 0 In fact, this is precisely the dal of or original linear program! Moreover, ineqality ( ) is jst an instantiation of the Weak Dality Theorem. I.6 Strong Dality The Fndamental Theorem can be rephrased in the folloing form: Strong Dality Theorem. If x is an optimal soltion for a canonical linear program Π, then there is an optimal soltion y for its dal, sch that c x = y Ax = y b. Proof (Sketch): I ll prove the theorem only for non-degenerate linear programs, in hich (a) the optimal soltion (if one exists) is a niqe vertex of the feasible region, and (b) at most d constraint planes pass throgh any point. These non-degeneracy assmptions are relatively easy to enforce in practice and can be removed from the proof at the expense of some technical detail. I ill also prove the theorem only for the case n d; the argment for nder-constrained LPs is similar (if not simpler). Let x be the optimal soltion for the linear program Π; non-degeneracy implies that this soltion is niqe, and that exactly d of the n linear constraints are satisfied ith eqality. Withot loss of generality (by permting the ros of A), e can assme that these are the first d constraints. So let A be the d d matrix containing the first d ros of A, and let A denote the other n d ros. Similarly, partition b into its first d coordinates b and everything else b. Ths, e have partitioned the ineqality Ax b into a system of eqations A x = b and a system of strict ineqalities A x < b. No let y = (y, y ) here y = ca 1 and y = 0. We easily verify that y b = c x : y b = y b + y b = y b = ca 1 b = c x. (The existence of the inverse matrix A 1 follos from or non-degeneracy assmption.) Similarly, it s easy to verify that y A c: y A = y A + y A = y A = c. 7

8 Once e prove that y is non-negative, and therefore feasible, the Weak Dality Theorem implies the reslt. We chose y = 0. As e ill see belo, the optimality of x implies the strict ineqality y > 0 e had to se optimality somehere! This is the hardest part of the proof. The key insight is to give a geometric interpretation to the vector y = ca 1. Each ro of the linear system A x = b describes a hyperplane a i x = b i in IR d. The vector a i is normal to this hyperplane and points ot of the feasible region. The vectors a 1,..., a d are linearly independent (by non-degeneracy) and ths describe a coordinate frame for the vector space IR d. The definition of y can be reritten as follos: c = y A = y i a i. In other ords, y lists the coefficients of the objective vector c in the coordinate frame a 1,..., a d. The point x lies on exactly d constraint hyperplanes; any d 1 of these hyperplanes determine a line throgh x. For each 1 i d, let l i denote the line that lies on all bt the ith constraint plane, and let v i denote a vector based at x that points into the halfspace a i x b i along the line l i. This vector lies along an edge of the feasible polytope. For all j i, e have a j v i = 0. Ths, e can rite i=1 A v i = (0,..., 0, a i v i, 0,..., 0) here the scalar a i v i appears in the ith coordinate. It follos that c v i = y A v i = y i (a i v i ). The optimality of x implies that c v i < 0, and becase v i points into the feasible region hile a i points ot, e have a i v i < 0. We conclde that y i > 0. We re done! Exercises 1. (a) Describe precisely ho to dalize a linear program ritten in slack form: s.t. Ax = b x 0 (b) Describe precisely ho to dalize a linear program ritten in general form: maximize c j x j sbject to a i j x j b i a i j x j = b i a i j x j b i for each i = 1.. p for each i = p p + q for each i = p + q n In both cases, keep the nmber of dal variables as small as possible. 8

9 2. A matrix A = (a i j ) is ske-symmetric if and only if a ji = a i j for all indices i j; in particlar, every ske-symmetric matrix is sqare. A canonical linear program max{c x Ax b; x 0} is self-dal if the matrix A is ske-symmetric and the objective vector c is eqal to the constraint vector b. (a) Prove any self-dal linear program Π is eqivalent to its dal program. (b) Sho that any linear program Π ith d variables and n constraints can be transformed into a self-dal linear program ith n + d variables and n + d constraints. The optimal soltion to the self-dal program shold inclde both the optimal soltion for Π (in d of the variables) and the optimal soltion for the dal program (in the other n variables). 3. (a) Model the maximm-cardinality bipartite matching problem as a linear programming problem. The inpt is a bipartite graph G = (U V ; E), here E U V ; the otpt is the largest matching in G. Yor linear program shold have one variable for each edge. (b) No dalize the linear program from part (a). What do the dal variables represent? What does the objective fnction represent? What problem is this!? 4. An integer program is a linear program ith the additional constraint that the variables mst take only integer vales. (a) Prove that deciding hether an integer program has a feasible soltion is NP-complete. (b) Prove that finding the optimal feasible soltion to an integer program is NP-hard. [Hint: Almost any NP-hard decision problem can be formlated as an integer program. Pick yor favorite.] 5. Helly s theorem states that for any collection of convex bodies in IR d, if every d + 1 of them intersect, then there is a point lying in the intersection of all of them. Prove Helly s theorem for the special case here the convex bodies are halfspaces. Eqivalently, sho that if a system of linear ineqalities Ax b does not have a soltion, then e can select d + 1 of the ineqalities sch that the reslting sbsystem also does not have a soltion. [Hint: Constrct a dal LP from the system by choosing a 0 cost vector.] c Copyright 2009 Jeff Erickson. Released nder a Creative Commons Attribtion-NonCommercial-ShareAlike 3.0 License ( Free distribtion is strongly encoraged; commercial distribtion is expressly forbidden. See for the most recent revision. 9