The Ellipsoid Algorithm

Transcription

1 The Ellipsoid Algorithm John E. Mitchell Department of Mathematical Sciences RPI, Troy, NY USA 9 February 2018 Mitchell The Ellipsoid Algorithm 1 / 28

2 Introduction Outline 1 Introduction 2 Assumptions 3 Each iteration 4 Complexity analysis 5 Separation and optimization 6 Relevance to integer optimization 7 N P-Hard problems Mitchell The Ellipsoid Algorithm 2 / 28

3 Introduction Introduction The simplex algorithm requires exponentially many iterations in the worst case for known pivot rules. In contrast, the ellipsoid algorithm requires a polynomial number of iterations. The ellipsoid algorithm was developed in the 1970s in the Soviet Union, building on earlier Soviet work on nonlinear programming. Mitchell The Ellipsoid Algorithm 3 / 28

4 Introduction New York Times, November 7, 1979 Mitchell The Ellipsoid Algorithm 4 / 28

5 Introduction New York Times, November 7, 1979 Published: November 7, 1979 Copyright The New York Times Mitchell The Ellipsoid Algorithm 5 / 28

6 Introduction New York Times, November 7, 1979 Published: November 7, 1979 Copyright The New York Times Mitchell The Ellipsoid Algorithm 6 / 28

7 Find a feasible solution Introduction The algorithm is a high-dimensional version of binary search. It can be stated as an algorithm to determine whether a feasible solution exists to a system of inequalities: Does there exist x 2 IR n satisfying a T i x apple c i for i = 1,...,m? We let P := {x 2 IR n : a T i x apple c i for i = 1,...,m}. Mitchell The Ellipsoid Algorithm 7 / 28

8 Assumptions Outline 1 Introduction 2 Assumptions 3 Each iteration 4 Complexity analysis 5 Separation and optimization 6 Relevance to integer optimization 7 N P-Hard problems Mitchell The Ellipsoid Algorithm 8 / 28

9 Assumptions Assumptions 1%4 We need to make two assumptions: Assumption 1: If P is nonempty then it includes points contained in a ball of radius R centered at the origin. Assumption 1 can be justified through the use of Cramer s Rule, which lets us bound the norm of any extreme point of P. Mitchell The Ellipsoid Algorithm 9 / 28

10 Assumption 2 Assumptions radius r X p Assumption 2: If P is nonempty then it contains a ball of radius r. Assumption 2 is like a nondegeneracy assumption. Cramer s Rule also lets us place a lower bound on the distance between two extreme points. Technically, we can always make Assumption 2, because the size of the numbers in the parameters can be used to define another parameter and then each c i can be increased by in such a way that the new system has a solution if and only if the old system has a solution, and the new system contains a ball of radius r if it is nonempty. Mitchell The Ellipsoid Algorithm 10 / 28

11 Assumptions Assumption 2 Assumption 2: If P is nonempty then it contains a ball of radius r. Assumption 2 is like a nondegeneracy assumption. Cramer s Rule also lets us place a lower bound on the distance between two extreme points. Technically, we can always make Assumption 2, because the size of the numbers in the parameters can be used to define another parameter and then each c i can be increased by in such a way that the new system has a solution if and only if the old system has a solution, and the new system contains a ball of radius r if it is nonempty. Mitchell The Ellipsoid Algorithm 10 / 28

12 Assumptions Assumption 2 Assumption 2: If P is nonempty then it contains a ball of radius r. Assumption 2 is like a nondegeneracy assumption. Cramer s Rule also lets us place a lower bound on the distance between two extreme points. Technically, we can always make Assumption 2, because the size of the numbers in the parameters can be used to define another parameter and then each c i can be increased by in such a way that the new system has a solution if and only if the old system has a solution, and the new system contains a ball of radius r if it is nonempty. Mitchell The Ellipsoid Algorithm 10 / 28

13 Each iteration Outline 1 Introduction 2 Assumptions 3 Each iteration 4 Complexity analysis 5 Separation and optimization 6 Relevance to integer optimization 7 N P-Hard problems Mitchell The Ellipsoid Algorithm 11 / 28

14 Each iteration Each iteration At each iteration we have an ellipsoid which contains (part of) P, if P is nonempty. We check the center x of the ellipsoid: does it satisfy If YES: we are done. a T i x apple c i for i = 1,...,m? Otherwise, we have a violated constraint a T i x apple c j. We define a new ellipsoid that contains the intersection of the old ellipsoid with the halfspace {x 2 IR n : ai T x apple ai T x}. If n = 1 then this would divide our search space in half. In IR n, we don t get such a good improvement. Nonetheless, the size of the ellipsoid shrinks enough that we get convergence in a number of iterations polynomial in n. Mitchell The Ellipsoid Algorithm 12 / 28

15 Each iteration Illustration: trying to find a point in P P x k E k Mitchell The Ellipsoid Algorithm 13 / 28

16 Each iteration Illustration: trying to find a point in P P x k E k a T i x = a T i x k Mitchell The Ellipsoid Algorithm 13 / 28

17 Each iteration Illustration: trying to find a point in P E "= {xer":(x-x" TY! (x-xmei} A P x k+1 x k positive definite, symmetric E k Volume: proportional todeft(mu) E k+1 a T i x = a T i x k Mitchell The Ellipsoid Algorithm 13 / 28

18 Each iteration Illustration: trying to find a point in P P x k+1 E k+1 Mitchell The Ellipsoid Algorithm 13 / 28

19 Each iteration Complexity analysis Initial ellipsoid: Ball of radius R centered at the origin. How many Noes are enough? If the ellipsoid is too small to contain a ball of radius r then P = ;. The algorithm updates from an ellipsoid E to an ellipsoid E 0. It can be shown that the volumes of these ellipsoids are related as: Vol(E 0 ) apple e 1/(2(n+1)) Vol(E). Mitchell The Ellipsoid Algorithm 14 / 28

20 Complexity analysis Outline 1 Introduction 2 Assumptions 3 Each iteration 4 Complexity analysis 5 Separation and optimization 6 Relevance to integer optimization 7 N P-Hard problems Mitchell The Ellipsoid Algorithm 15 / 28

21 How many iterations? Complexity analysis How many Noes to go from a ball of radius R to a ball of radius r? Let v and V denote the volumes of balls of radius r and R, respectively. Let E 0 be the initial ellipsoid, let E k be the ellipsoid after k iterations. We want the smallest value of k so that Vol(E k ) apple v. We also have Vol(E k ) apple h e 1/(2(n+1))i k V = e k/(2(n+1)) V. Thus it suffices to choose k sufficiently large that e k/(2(n+1)) V apple v. Mitchell The Ellipsoid Algorithm 16 / 28

22 Complexity analysis How many iterations, part 2 It suffices to choose k sufficiently large that e k/(2(n+1)) V apple v. Equivalently, after dividing by V and taking logs of both sides, we require v k/(2(n + 1)) apple ln, V which can be restated as requiring k 2(n + 1) ln V. v The ratio V /v can be chosen so that ln(v /v) is O(n 3 ), so we have an algorithm that requires O(n 4 ) iterations. Mitchell The Ellipsoid Algorithm 17 / 28

23 Work per iteration Complexity analysis In exact arithmetic, would need to work with square roots. Can show that it is OK to approximate the square roots and still have the volume of the ellipsoid decrease at almost the same rate. Mitchell The Ellipsoid Algorithm 18 / 28

24 Complexity analysis Finding optimal solutions to linear programs If feasible, cut on the objective function. So restrict attention to the halfspace of points at least as good as the current center. Mitchell The Ellipsoid Algorithm 19 / 28

25 Separation and optimization Outline 1 Introduction 2 Assumptions 3 Each iteration 4 Complexity analysis 5 Separation and optimization 6 Relevance to integer optimization 7 N P-Hard problems Mitchell The Ellipsoid Algorithm 20 / 28

26 Separation and optimization Convex feasibility problems = a Tx< c I a t >a C Definition Given a convex set C IR n and a point x 2 IR n, the separation problem is to determine whether x 2 C. If the answer is NO, a separating hyperplane a T x = c is determined, where a 2 IR n,c2 IR, a T x apple c for all x 2 C, and a T x > c. Mitchell The Ellipsoid Algorithm 21 / 28

27 Separation and optimization Equivalence of separation and feasibilty Theorem Assume the convex set C contains a ball of radius r. If the separation problem for C can be solved in polynomial time then the feasibility problem for C can also be solved in polynomial time. Proof. We use the ellipsoid algorithm. At each iteration, we test whether the center x k of the current ellipsoid is in C. If it is not then the separation problem solution returns a valid constraint for C which we can use to update the ellipsoid. The overall complexity is the product of two polynomial functions, which is polynomial. Mitchell The Ellipsoid Algorithm 22 / 28

28 Separation and optimization Equivalence of separation and feasibilty Theorem Assume the convex set C contains a ball of radius r. If the separation problem for C can be solved in polynomial time then the feasibility problem for C can also be solved in polynomial time. Proof. We use the ellipsoid algorithm. At each iteration, we test whether the center x k of the current ellipsoid is in C. If it is not then the separation problem solution returns a valid constraint for C which we can use to update the ellipsoid. The overall complexity is the product of two polynomial functions, which is polynomial. Mitchell The Ellipsoid Algorithm 22 / 28

29 " i

30 Separation and optimization Equivalence of separation and optimization Consider the optimization problem min x2ir n c T x subject to x 2 C IR n (CP) where C is a convex set and c 2 IR n. Theorem The problem (CP) can be solved in polynomial time if and only if the separation problem for C can be solved in polynomial time. This result follows from using the ellipsoid algorithm. We can optimize over C using the ellipsoid algorithm, generating separating hyperplanes either: by solving the separation problem to separate the current center of the ellipsoid from C, or by cutting on the objective function if x 2 C. Mitchell The Ellipsoid Algorithm 23 / 28

31 Separation and optimization Equivalence of separation and optimization Consider the optimization problem min x2ir n c T x subject to x 2 C IR n (CP) where C is a convex set and c 2 IR n. Theorem The problem (CP) can be solved in polynomial time if and only if the separation problem for C can be solved in polynomial time. This result follows from using the ellipsoid algorithm. We can optimize over C using the ellipsoid algorithm, generating separating hyperplanes either: by solving the separation problem to separate the current center of the ellipsoid from C, or by cutting on the objective function if x 2 C. Mitchell The Ellipsoid Algorithm 23 / 28

32 Separation and optimization Equivalence of separation and optimization Consider the optimization problem min x2ir n c T x subject to x 2 C IR n (CP) where C is a convex set and c 2 IR n. Theorem The problem (CP) can be solved in polynomial time if and only if the separation problem for C can be solved in polynomial time. This result follows from using the ellipsoid algorithm. We can optimize over C using the ellipsoid algorithm, generating separating hyperplanes either: by solving the separation problem to separate the current center of the ellipsoid from C, or by cutting on the objective function if x 2 C. Mitchell The Ellipsoid Algorithm 23 / 28

33 ±. Ex<<7 4

34 Relevance to integer optimization Outline 1 Introduction 2 Assumptions 3 Each iteration 4 Complexity analysis 5 Separation and optimization 6 Relevance to integer optimization 7 N P-Hard problems Mitchell The Ellipsoid Algorithm 24 / 28

35 Relevance to integer optimization Integer optimization as a linear program Consider the integer optimization problem min x c T x subject to x 2 S (IP) x 2 Z n IR n for some subset S, and where Z denotes the set of integers. Let C denote the convex hull of the set of integer points in S, so C = conv(s \ Z n ) (The convex hull of a set T is the smallest convex set containing T.A formal definition will be given next week.) All the extreme points of C are feasible integer points, so we could in principle solve (IP) by solving the linear program min x c T x subject to x 2 C IR n (LP). Mitchell The Ellipsoid Algorithm 25 / 28

36 %t I.

37 Relevance to integer optimization Solving the linear program The difficulty with this approach is that it is very hard to get an explicit formulation for C. If we can solve the separation problem for C in polynomial time then we can solve (LP) in polynomial time, which means we can solve (IP) in polynomial time. Thus, for N P-complete problems, we cannot expect to solve the separation problem in polynomial time. Conversely, if we can solve the optimization problem in polynomial time then we can solve the separation problem in polynomial time. For example, we can solve the problem of finding a minimum weight perfect matching on a graph G =(V, E) in polynomial time, and indeed the separation problem of determining whether a point x is in the convex hull of the set of perfect matchings can be solved in O( V 4 ) time. Mitchell The Ellipsoid Algorithm 26 / 28

38 N P-Hard problems Outline 1 Introduction 2 Assumptions 3 Each iteration 4 Complexity analysis 5 Separation and optimization 6 Relevance to integer optimization 7 N P-Hard problems Mitchell The Ellipsoid Algorithm 27 / 28

39 N P-hard problems N P-Hard problems A problem is called N P-hard if it at least as hard as an N P-complete problem, in the sense that the N P-complete problem can be polynomially reduced to the N P-hard problem. An N P-hard problem need not be a feasibility problem. In particular, optimization versions of N P-complete integer programming feasibility problems are N P-hard. Mitchell The Ellipsoid Algorithm 28 / 28