Preliminaries and Complexity Theory Oleksandr Romanko CAS 746 - Advanced Topics in Combinatorial Optimization McMaster University, January 16, 2006
Introduction Book structure: 2
Part I Linear Algebra : Introduction: Book Structure theory of linear spaces and of systems of linear equations complexity of solving linear systems sizes of solutions and the running time of the Gaussian elimination method which is a polynomial-time one (its running time is bounded by a polynomial in the size of the input data) Part II Lattices and Linear Diophantine Equations : solving systems of linear diophantine equations (solve systems of linear equations in integer variables) Hermite normal form problem of diophantine approximation (the continued fraction method approximates a real number by a rational number with low denominator) 3
Introduction: Book Structure Part III Polyhedra, Linear Inequalities, and Linear Programming : solving systems of linear inequalities solving systems of linear equations in nonnegative variables solving linear programming problems Polynomial-time Part IV Integer Linear Programming : solving systems of linear diophantine inequalities, i.e. solving linear inequalities in integers solving systems of linear equations in nonnegative integer variables solving integer linear programming problems NP-complete 4
If a is a real number, then General Preliminaries denote the lower integer part and the upper integer part of α. Notation: - sets of integers, rationals, and real numbers - sets of nonnegative integers, rationals, and real numbers Notation: Greatest common divisor: If are rational numbers (not all equal to 0) then the largest rational number dividing each of exists, and is called the greatest common divisor or g.c.d. The numbers are relatively prime if 5
General Preliminaries Big O notation is used to describe the asymptotic behavior of functions - an asymptotic upper bound for the magnitude of a function in terms of another, usually simpler, function Optimization problem: 6
Preliminaries: Linear Algebra, Linear equations and inequalities: Matrix Theory, Geometry A system of linear inequalities can have several alternative forms: 7
Euclidian norm: Preliminaries: Linear Algebra, Matrix Theory, Geometry Euclidian ball: Other norms: 8
Matrices: Preliminaries: Linear Algebra, Row (column) rank Row (column) submatrix Upper and low triangular matrices Diagonal matrices Matrix Theory, Geometry Incidence vector (characteristic vector) of a set T S (universe S) is a {0,1} vector χ T whose entries are labeled with the members of S: The support of a vector is the set of coordinates at which the vector is non-zero 9
Preliminaries: Linear Algebra, A set C of vectors is convex if it satisfies: Matrix Theory, Geometry Linear hull, affine hull and convex hull of a set X of vectors. Convex hull is the smallest convex set containing X. A (convex) cone is a nonempty set of vectors C satisfying The cone generated by a set X of vectors is the smallest convex cone containing X 10
Preliminaries: Pivoting 11
Preliminaries: Some Inequalities Cauchy-Schwarz inequality: Hadamard inequality (B is the m x n matrix consisting of column vectors b 1,,b m ): 12
Preliminaries: Positive Definite Matrices A real-valued matrix D is called positive definite if D is symmetric and its eigenvalues all are positive. The following are equivalent for a realvalued symmetric matrix D: Let D be a positive definite matrix, and let D = C T C, for nonsingular C. Two vectors c and d are orthogonal (relative to the inner product defined by D) if c T Dd = 0. The norm. defined by D is given by 13
Preliminaries: Balls and Ellipsoids 14
Graph Theory: Undirected Graphs Undirected graph is a pair G = (V, E), where V is a finite set, and E is a family of unordered pairs of elements of V. The elements of V are called the vertices or points of G, and the elements of E are called the edges or lines of G. A pair occurring more than once in E is a multiple edge. Graphs without multiple edges are called simple. Loops are edges of the form {v,v} We say that an edge {v,w} connects the vertices v and w The vertices v and w are adjacent if there is an edge connecting v and w The edge {v,w} is incident with the vertex v and w, and conversely The vertices v and w are called the ends of the edge {v,w} The number of edges incident with a vertex v is called the valency or degree of, usually denoted by d G (v). The maximum and minimum degree of the vertices of G are denoted by (G) and δ(g) 15
Graph Theory: Undirected Graphs Example: G=(V,E), V(G)={u,v,w,x} and E(G)={{u,v}.{v,w},{v,x},{w,x}} e u v w f v and w are the ends (endpoints) of the edge e={v,w} e is incident with both v and w w is incident with the edges e and f={w,x} The vertex x is adjacent to the vertices v and w The edge {v,w} is adjacent to the edges {u,v}, {v,w} and {w,x} The degree of v is d G (v)=3 The maximum and minimum degree of the vertices of G are (G)=3 and δ(g)=1 x 16
Graph Theory: Undirected Graphs The complementary graph of G is the simple graph with the same vertex set as G, and with edges all pairs {v, w} of vertices which are not in E. A simple graph is complete if E is the set of all pairs of vertices (K n ). A graph G = (V,E ) is a subgraph of G = (V,E) if V V and E E. A planar graph is one that can be drawn on a plane in such a way that there are no "edge crossings," i.e. edges intersect only at their common vertices. Example: K 4 is complete planar graph, K 5 and K 3,3 are complete nonplanar graphs K 4 K 5 K 3,3 17
Graph Theory: Undirected Graphs A path in the graph G=(V,E) from v 0 to v t is an alternating sequence of vertices and edges of the form If v 0 = v t, we have closed path. A graph is connected if each two vertices of the graph are connected by a path 18
Graph Theory: Undirected Graphs Algorithm for finding the distance between vertices r and s. It consists of recursively determining the sets V 0, V 1, V 2,..., where V i, is the set of vertices v of distance i to r : 19
Graph Theory: Undirected Graphs A closed path (v 0 = v t ) of length at least one and without repeated edges or vertices (except for the end points) is called a circuit. A Hamilton circuit is a circuit containing each vertex of the graph exactly once (except for the end points). Famous traveling salesman problem (TSP): given a graph G = (V,E) and a 'length' function f, find a Hamilton circuit of minimum length. (The length of a circuit is the sum of the lengths of its edges.) 20
Graph Theory: Undirected Graphs A forest is a graph having no circuits, and a tree is a connected forest Equivalently, a tree is a graph in which any two vertices are connected by exactly one path. A forest is a graph in which any two vertices are connected by at most one path. Forests Trees 21
Graph Theory: Undirected Graphs A clique (coclique, respectively) in the graph G = (V,E) is a set of pairwise adjacent (nonadjacent, respectively) vertices of G. The (maximum) clique problem is the problem: given a graph G = (V,E), find a clique of maximum size A clique of a graph is its maximal complete subgraph!!! A matching is a set of pairwise disjoint edges (a set of edges of G such that no two of them share a vertex in common). A matching covering all vertices is called a perfect matching. 22
Graph Theory: Undirected Graphs A graph G = (V,E) is called bipartite if V can be partitioned into two classes V 1 and V 2 such that each edge of G contains a vertex in V 1 and a vertex in V 2. The sets V 1 and V 2 are called colour classes. A complete bipartite graph is a simple bipartite graph, with colour classes V 1 and V 2, so that each vertex in V 1, is adjacent to each vertex in V 2. A complete bipartite graph with colour classes of sizes m and n is denoted by K m,n. K 3,3 23
Graph Theory: Undirected Graphs The adjacency matrix of an undirected graph G = (V,E) is the matrix with both rows and columns indexed by V, where the entry in position (v, w) is the number of edges connecting v and w. So G is simple if and only if its adjacency matrix is a {0,1}-matrix. The incidence matrix of G is the {0, 1}-matrix with rows and columns indexed by V and E, respectively, where the entry in position (v, e) is 1 if and only if vertex v is incident with edge e. 24
Graph Theory: Directed Graphs A directed graph or a digraph is a pair D = (V, A), where V is a finite set, and A is a finite family of ordered pairs of elements of V. The elements of V are called the vertices or points, and the elements of E are called the arcs of D. The vertices v and w are called the tail and the head of the arc (v, w), respectively. Arc (v, w) enters w and leaves v If W is a set of vertices such that v W and w W, then (v,w) is said to enter W and to leave V\ W. If W V, then δ A (W) or δ (W) denotes the set of arcs in A entering W, and δ + A (W) or δ+ (W) denotes the set of arcs in A leaving W 25
Graph Theory: Directed Graphs Given a digraph D = (V, A) and vertices r,s V, a flow from r to s, or an r-s-flow, is a function x:a R satisfying: The second set of constraints here means that at any vertex v r, s, the total 'amount' of flow entering v is equal to the total 'amount' of flow leaving v the flow conservation law. The value of the flow is the net amount of flow leaving r is equal to the net amount of flow entering s. Flow x is said to be subject to a given 'capacity' function c:a R + if x(a) c(a) for each arc a. A circulation is a flow for which the flow conservation law holds at each vertex (i.e. it is an r-s-flow of value 0, or an r-r-flow, for arbitrary r and s). 26
Graph Theory: Directed Graphs An undirected path (undirected circuit, respectively) for a directed graph is a path (circuit, respectively) in the underlying undirected graph. An undirected path or circuit in a directed graph has forward arcs and backward arcs. A digraph D = (V, A) is called strongly connected if for each two vertices r and s of D there is a directed r-s-path. D is (weakly) connected if its underlying undirected graph is connected. The incidence matrix of a digraph D = (V, A) is the matrix with rows and columns indexed by V and A, respectively, where the entry in position (v,a) is -1, +1, or 0, if vertex v is the head of a, the tail of a, or neither, respectively. 27
Problems, Algorithms and Complexity: Letters, Words and Sizes Need symbols and strings of symbols for formalizing problem complexity. Let Σ be a finite set (often Σ = {0,1}). Σ is called the alphabet and its elements are called symbols or letters. An ordered finite sequence of symbols from Σ is called a string {of symbols) or a word. Σ* stands for the collection of all strings of symbols from Σ. The size of a string is the number of its components. The string of size 0 is the empty string, denoted by. Strings can have the form of (finite sequences of) rational numbers, vectors, etc. Standard way of encoding strings of symbols is binary notation (fixed alphabet like {0,1}). Fix the sizes of a rational number (where p and q are relatively prime integers), of a rational vector and of a rational matrix are 28
Problems, Algorithms and Complexity: Letters, Words and Sizes Strings can have the form of (finite sequences of) rational numbers, vectors, etc. Standard way of encoding strings of symbols is binary notation (fixed alphabet like {0,1}). Fix the sizes of a rational number (where p and q are relatively prime integers), of a rational vector and of a rational matrix are 29
Problems, Algorithms and Complexity: Problems A problem can have the form of a question or a task. Mathematically, a (search) problem is a subset Π of Σ* Σ*, where Σ is some alphabet. The corresponding problem then is: Here the string z is called an instance or the input of the problem, and y is a solution or the output. Problem Π is called a decision problem or a yes/no problem if, for each (z,y) in Π, y is the empty string. In that case, the problem is often identified with the set if of strings z in Σ* for which (z, ) belongs to Π. The problem can have form like 30
Problems, Algorithms and Complexity: Examples of search problems: Problems Corresponding informal forms of these problems: 31
Problems, Algorithms and Complexity: Algorithms and Running Time An algorithm can be formalized in terms of Turing Machine (TM). For a given input z Σ*, an algorithm for problem Π Σ* Σ* determines an output y such that (z,y) is in Π, or stops without delivering an output if there exists no such y. Mathematically, an algorithm is a finite string A of 0's and 1's. A solves problem Π or A is an algorithm for Π, if for any instance z of Π, when giving the string (A,z) to a 'universal Turing machine', the machine stops after a finite number of steps, while delivering a string y with (z,y) Π, or delivering no string in the case where such a string y does not exist. Running time - the number of 'elementary bit operations' in the execution of an algorithm. Mathematically, the running time of an algorithm A for a certain problem instance z is the number of moves the 'head' of a universal Turing machine makes before stopping when given the input (A,z). Define the running time (function) of an algorithm A as the function 32
Problems, Algorithms and Complexity: Polynomial Algorithms Complicated definition A function f(n) is said to be polynomially bounded if f(n) = O(n k ). Easier definition 33
Problems, Algorithms and Complexity: Polynomial Algorithms A function f(n) is said to be polynomially bounded if f(n) = O(n k ). An algorithm is polynomial-time or polynomial if its running time function is polynomially bounded. A problem is said to be solvable in polynomial time or polynomially solvable if the problem can be solved by a polynomial-time algorithm. We are interested mostly in the asymptotic behavior of the running time of an algorithm. Therefore, we say that the running time is O(g(σ)), for some function g(σ), meaning that there is a constant C such that the running time is upper bounded by Cg(σ). The elementary arithmetic operations are: adding, subtracting, multiplying, dividing, and comparing numbers. In rational arithmetic, they can be executed by polynomial-time algorithms. Therefore, for deriving the polynomiality of an algorithm which performs a sequence of elementary arithmetic operations, it suffices to show that the total number of these operations is polynomially bounded by the size of the input, and that the sizes of the intermediate numbers to which these 34 elementary arithmetic operations are applied are polynomially
Problems, Algorithms and Complexity: the Classes P, NP and co-np The class of decision problems solvable in polynomial time is denoted by P. Another, possibly larger, 'complexity class' is the class NP. Informally, the class NP can be described as the class of decision problems satisfying Formally, a decision problem belongs to NP if there exists a polynomially solvable decision problem and a polynomial such that for each z in Σ*: 35
Problems, Algorithms and Complexity: the Classes P, NP and co-np Formally, a decision problem belongs to NP if there exists a polynomially solvable decision problem and a polynomial such that for each z in Σ*: Interpretation: y here fulfils the role of a 'polynomial-length proof of the fact that z is in. This 'proof can be checked in polynomial time, as is polynomially solvable. It is not required that y in is to be found in polynomial time. The string y can be seen as a 'certificate' for z, which can be given to the 'boss' in order to convince him that z is in. To test whether z belongs to, we could 'guess' a string y such that (z,y) is in. Since there are at most such guesses, it follows that if is in NP, there is a polynomial and an algorithm which solves the problem in time, where is the input size. 36
Problems, Algorithms and Complexity: the Classes P, NP and co-np The complement of a decision problem is the decision problem. The class of decision problems whose complement is in NP is denoted by co-np. So, co-np consists of those decision problems for which the fact that a certain string z is not in has a proof of length polynomially bounded by size(z). Since the complement of every polynomially solvable decision problem is polynomially solvable again, we know that P co-np, and hence P NP co-np. The class NP co-np consists of those decision problems for which both a positive answer and a negative answer have a proof of polynomial length. That is, it consists of all problems for which there exist polynomially solvable decision problem and a polynomial, such that for each string z Σ*: 37
Problems, Algorithms and Complexity: the Classes P, NP and co-np It is as yet a big open problem in complexity theory whether the class P is equal to the class NP. It is generally believed that P NP. The traveling salesman problem is one of the many problems in NP for which no polynomial-time algorithm has been found as yet. Most of these problems turned out to belong to the hardest in NP, the socalled NP-complete problems. Similarly, it is unknown whether NP = co-np or P = NP co-np. Thus we have the following Venn diagram: 38
Problems, Algorithms and Complexity: NP-Complete Problems Certain problems in the class NP are the hardest among all problems in NP. Among these hardest problems are the integer linear programming problem, the traveling salesman problem, and the maximum clique problem. A decision problem is called Karp-reducible to decision problem if there is a polynomial-time algorithm A such that for any input string z in Σ*, A delivers a string x as output, such If is polynomially solvable, and is Karp-reducible to, then also is polynomially solvable. Similarly, if belongs to NP, and is Karp-reducible to, then also belongs to NP. The same applies to co-np. 39
Problems, Algorithms and Complexity: NP-Complete Problems 40