Farkas Lemma Rudi Pendavingh Eindhoven Technical University Optimization in R n, lecture 2 Rudi Pendavingh (TUE) Farkas Lemma ORN2 1 / 15
Today s Lecture Theorem (Farkas Lemma, 1894) Let A be an m n matrix, b R m. Then either: 1 There is an x R n such that Ax b; or 2 There is a y R m such that y 0, ya = 0 and yb < 0. Farkas Lemma is at the core of linear optimization. There are extremely efficient proofs of Farkas Lemma. We ll do a slow, geometrical proof. Farkas Lemma is augmented by Carathéodory s theorem. Rudi Pendavingh (TUE) Farkas Lemma ORN2 2 / 15
Convex sets Definition Let x, y R n. The line segment between x and y is [x, y] := {λx + (1 λ)y λ [0, 1]}. Definition A set C R n is convex if [x, y] C for all x, y C. Rudi Pendavingh (TUE) Farkas Lemma ORN2 3 / 15
Lemma If C α R n is convex for each α, then α C α is convex. A map f : R m R n is affine if f : x Ax + b for some A, b. Lemma If C R n is convex and f : R m R n is affine, then f 1 [C] is convex. Example any affine space {x R n Ax = b} is convex any halfspace {x R n ax β} is convex any polyhedron {x R n Ax b} is convex the unit ball {x R n x 1} is convex any ellipsoid {Ax + b x 1} (det(a) 0) is convex Rudi Pendavingh (TUE) Farkas Lemma ORN2 4 / 15
Separation Definition A set H R n is a hyperplane if H = H d,δ := {x R n dx = δ} for some nonzero rowvector d R n and δ R. H < H > H Definition Let X, Y R n. Then H d,δ separates X from Y if dx < δ for all x X, and dy > δ for all y Y. Rudi Pendavingh (TUE) Farkas Lemma ORN2 5 / 15
The Separation Theorem Theorem Let C R n be a closed, convex set and let x R n. If x C, then there is a hyperplane separating {x} from C. Proof. 1 min{ y x y C} is attained; let z C attain the minimum. 2 Let d := (z x) t and δ := 1 2d(z + x). 3 H d,δ separates {x} from C: dx < δ and dy > δ for all y C Rudi Pendavingh (TUE) Farkas Lemma ORN2 6 / 15
Separation variations Theorem Let C be an open convex set and let x R n \ C. Then there exists a hyperplane H such that x H and C H =. Proof. Let x i R n \ C be such that lim i x i = x. Apply the Separation theorem to C, x i. Construct H. Theorem Let C, D R n be disjoint open convex sets. Then there is a hyperplane H separating C from D. Proof... up to you. Rudi Pendavingh (TUE) Farkas Lemma ORN2 7 / 15
Cones Definition A set C R n is a cone if αx + βy C for all x, y C and α, β > 0. Example The Lorenz cone is L n := {x R n x 2 1 + + x 2 n 1 x n}. Example Let a 1,..., a m R n. Then the cone spanned by a 1,..., a m is cone{a 1,..., a m } := {λ 1 a 1 + + λ m a m λ i 0 for all i}. Rudi Pendavingh (TUE) Farkas Lemma ORN2 8 / 15
Farkas Lemma Theorem Let C R n be a closed cone and let x R n. Either 1 x C, or 2 there is a d R n such that dy 0 for all y C and dx < 0. Theorem (Farkas Lemma, 1894) Let a 1,..., a m, b R n. Then either 1 b cone{a 1,..., a m }; or 2 there is a d R n such that da i 0 for all i and db < 0. Lemma Let a 1,..., a m R n. Then cone{a 1,..., a m } is a closed set. Rudi Pendavingh (TUE) Farkas Lemma ORN2 9 / 15
Farkas Lemma variations Theorem (Farkas Lemma, variant 1) Let A be an m n matrix, let b R m. Then either: 1 there is an x R n such that Ax = b and x 0; or 2 there is a y R m such that ya 0 and yb < 0. Theorem (Farkas Lemma, variant 2) Let A be an m n matrix, b R m. Then either: 1 there is an x R n such that Ax b; or 2 there is a y R m such that y 0, ya = 0 and yb < 0. Proof. Apply the previous theorem to A := [ A A I ], b := b. Rudi Pendavingh (TUE) Farkas Lemma ORN2 10 / 15
Farkas Lemma variations Definition If L R n is a linear space, then is the orthogonal complement of L. L := {y R n y x for all x L} Theorem (Farkas Lemma, coordinate-free variant) Let L R n be a linear space. Exactly one of the following holds: 1 there exists an x L such that x 0 and x n > 0; or 2 there exists an y L such that y 0 and y n > 0. Proof. [ x Without loss of generality L = { t ] Ax = bt}. Apply Variant 1. Rudi Pendavingh (TUE) Farkas Lemma ORN2 11 / 15
Carathéodory s theorem Theorem (Carathéodory, 1911) Let S R n be a finite set, and let x R n. If x cone S then there is a linearly independent set T S such that x conet. Proof. 1 Let T S be the smallest set such that x conet. 2 T is linearly independent: if not, there are µt R (not all 0) such that t T µ tt = 0. there are λt 0 such that t T λ tt = x. there is an α R such that λt + αµ t 0 for all t, and λ t0 + αµ t0 = 0 for some t 0. then x cone(t \ {t 0 }); contradiction. Rudi Pendavingh (TUE) Farkas Lemma ORN2 12 / 15
Few linear inequalities suffice for inconsistency Theorem (Carathéodory, 1911) Let S R n be a finite set, and let x R n. If x cone S then there is a linearly independent set T S such that x conet. Corollary If the system of linear inequalities a 1 x b 1,..., a m x b m, has no solution x R n, then there is a set J {1,..., m} with at most n + 1 members such that the subsystem has no solution x R n. a i x b i for all i J Rudi Pendavingh (TUE) Farkas Lemma ORN2 13 / 15
Linear inequalities have extreme solutions Theorem (Carathéodory, 1911) Let S R n be a finite set, and let x R n. If x cone S then there is a linearly independent set T S such that x conet. Corollary If the system of linear inequalities a 1 x b 1,..., a m x b m, has a solution x R n, then there also is a solution x such that lin.hull{a 1,..., a m } = lin.hull{a i a i x = b i }. Rudi Pendavingh (TUE) Farkas Lemma ORN2 14 / 15
The fundamental theorem of linear inequalities Theorem Let a 1,..., a m, b R n. Then exactly one of the following holds: 1 there is a linearly independent subset T {a 1,..., a m } such that b cone T ; and 2 there is a nonzero d R n such that da i 0 for all i, db < 0, and rank{a i da i = 0} = rank{a 1,..., a m, b} 1. Rudi Pendavingh (TUE) Farkas Lemma ORN2 15 / 15