Algorithms and Architectures II H-P Schwefel, Jens M. Pedersen Mm6 Advanced Graph Algorithms (hps) Mm7 Intro to distributed computing (jmp) Mm8 Backtracking, 2-player games, genetic algorithms (hps) Mm9 Complex Problems in Network Planning (JMP) Mm10 Complexity Theory, Task scheduling examples hps@kom.auc.dk http://www.kom.auc.dk/~hps Page 1 Complexity: Some examples Graph Theory Finding shortest paths between all node pairs in a graph: Floyd Algorithm, time O(N^3), space O(N^2), N= Number of Nodes Find shortest paths from specific node to all destinations: Dijkstra Algorithm, time O(N^2) Search & Optimization Problems: Binary search in ordered set: O(log(N)) Ordering set of N elements: O(N log (N)) Traveling Salesman problem (shortest tour between N cities): Exhaustive Search, time O(N!) Finding an allocation of the N boolean variables x 1,...,x N such that the boolean function f(x 1,...,X N )=true: exhaustive search, time O(2^N) Finding the prime-factors of a large composite n=p*q with p,q=prime Interesting Questions: Can these (and other) problems be solved faster? Are there problems that cannot be computed by an algorithm at all? Page 2
Content 1. Motivation: Example Algorithms 2. Computation models Register Machines, Turing Machine Church s Thesis Non-determinism 3. Main sequential complexity classes Uncomputable problems P time, NP time, P space, NP space P-time and log-space reductions NP complete problem examples P complete problems examples 4. Task scheduling examples 5. Summary/Exercises Page 3 Computational Models & Computability Finite program (normally represented as string over some alphabet) Finite number of registers Assignments and basic arithmetics: Ri=0, Ri=Rj+Rk, INC/DEC Ri Different types of repetitive instructions Loop Ri While Ri 0 If Ri 0 goto label Recursion Tail recursion/primitive recursion μ-recursion/partial recursion Loop computable functions = Tail-recursive functions While-computable functions = if-goto computable functions = partial recursive functions Page 4
Computational Models: Turing Machine Informal Definition Turing Machine = automaton with finite state-set working on an infinite storage organized as read/write tape Operation: read one symbol from the current position on the tape; depending on read symbol and current state determine New state New symbol to be written at the current tape position Movement of current position to right or left For certain states and certain input symbols, the Turing machine may stop operation Turing Machine <Q,Γ,b,X, q 0,F,δ> Finite State-space Q Finite working alphabet Γ Empty symbol b Finite input Alphabet X Γ/{b} Starting state q 0 Q F Q is set of accepting final states Transition function δ: (Q/F) x Γ Γ x {L,R} x Q Page 5 Turing Machines & Computability Turing machines (cntd) Example: Addition of unary numbers Equally power-ful models (with respect to computability) Only one-sided infinite tape Multiple tapes Multiple read/write heads Multiple dimensions Turing computable = if-goto computable (extended on X* = set of sequences over X) Church Thesis: Any algorithmic computable function is Turing computable Page 6
Acceptance, decidability; diagonalisation L X* is called Turing acceptable (recursive enumerable), iff there is a TM M that Stops on all inputs v L in an acceptance state q f F For all v L, M may or may not stop, but if it stops, then in a state F L X* is called decidable (recursive), if there is a TM M that Stops on all inputs v X* but only accepts if v L Corollaries: L decidable X*/L decidable L acceptable and X*/L acceptable L decidable Diagonalisation: Let <M> be a binary encoding of the Turing Machine M Hence, all possible TMs can be enumerated w i =<M i > L d := { w i w i L(M i )}, there does not exist any TM that accepts L d Page 7 Non-determinism Non-deterministic Turing Machine M nd =<Q,Γ,b,X, q 0,F,τ> Finite State-space Q Finite working alphabet Γ Empty symbol b Finite input Alphabet X Γ/{b} Starting state q 0 Q F Q is set of accepting final states Transition function τ: (Q/F) x Γ (Γ x {L,R} x Q) Consequence: At each computation step, multiple (but finite) number of choices v X* accepted by M nd iff there exists one computation in the computation tree that accepts v L X* acceptable by a deterministic TM L acceptable by a non-determinstic TM Page 8
Content 1. Motivation: Example Algorithms 2. Computation models Register Machines, Turing Machine Church s Thesis Non-determinism 3. Main sequential complexity classes Uncomputable problems P time, NP time, P space, NP space P-time and log-space reductions NP complete problem examples P complete problems examples 4. Task Scheduling Example Problems 5. Summary/Exercises Page 9 Un-computable Problems Examples: L NE :={ w i L(M i ) } is Turing acceptable (recursive enumerable) L E :={ w i L(M i ) = } is not Turing decidable (recursive), hence also not Turing acceptable L h :={<M,w> M stops on input w}; L nh :=X*/L h ={<M,w> M does not stop on input w} L h is Turing acceptable but not decidable, L nh is not Turing acceptable It is not decidable whether a context-free grammar creates a regular language It is not decidable whether two context-free grammars have a non-empty intersection Page 10
Resources: Time and Space complexity Computational Model: Deterministic Turing Machine M with k one-sided infinite tapes Read-only tape that contains input of length n If for all inputs of length n M uses at most S(n) fields on any of the k tapes M has space complexity S(n) M uses at most T(n) steps M has time complexity T(n) Example: L={wbw R w {0,1}*} (Palindromes) Space complexity S(n)=log2(n) (using a binary counter) Time complexity T(n)=n+1 (while using space (n-1)/2) Page 11 Non-deterministic classes and relations L NSpace(S(n)): Iff there exists a non-deterministic TM that decides membership of L with no computation sequence exceeding S(n) fields on any tape L NTime(T(n)): Iff there exists a non-determinstic TM that decides membership of L with no computation requiring more than T(n) steps Similar Complexity classes for deterministic case: Dspace(S(n)), Dtime(T(n)) Relations (by definition): DTime(T(n)) NTime(T(n)) DTime(f(n)) DSpace(f(n)), NTime(f(n)) NSpace(f(n)) DSpace(S(n)) NSpace(S(n)) Reverse Directions: L DSpace(f(n)) and f(n)>log 2 (n) there exists a c>0 with L DTime(c f(n) ) L NTime(T(n)) there exists a c>0 with L DTime(c T(n) ) Page 12
Space compression and linear speed-up Due to freedom of choice of state-space and working alphabet (Notation: x=n or x=d, but consistently replaced within one line) Linear factors in complexity classes L xspace(s(n)) L xspace(c*s(n)), for all c>0 L xtime(t(n)), and lim T T(n)/n= L xtime(c*t(n)) for all c>0 L xtime(c*n) L xtime((1+ε)*n) for all ε>0 Reduction from k-tape TM L xspace(s(n)) for k tapes L xspace(s(n)) for TM with one tape L xtime(t(n)) for k>1 tapes L xtime(t 2 (n)) for single tape TM L xtime(t(n)) for k>2 tapes L xtime(t(n) log T(n)) for TM with 2 tapes Page 13 Polynomial time and space: P, NP, PSpace Time Efficient Problems: Polynomial Time P P := PTime := i 1 DTime(n i ) Non-deterministic polynomial Time: NP NP := NPtime := i 1 NTime(n i ) Polynomial Space: Pspace PSpace := i 1 DSpace(n i ) Relation: DSpace(log n) P NP PSpace And it is known: DSpace(log n) PSpace, hence at least one of the relations is a true subset in the chain above, but it is not known which one(s) Page 14
Reductions, hard and complete problems Definition: L < g L, L is reducible to L via the function g iff x L g(x) L For complexity investigations: resource limited reductions Polynomial time reductions < p : g(x) computable in deterministic polynomial time Log-Space reductions < l : g(x) computable using deterministic logarithmic space L is called NP-hard, iff for all L NP, L < p L L is called NP-complete, iff L NP-hard and L NP Examples of NP complete problems: SAT: Do the variables X 1,...,X n in a boolean expression have an allocation such that the expression=true? (Directed) Hamilton Circles: Is there a path in the (directed) graph G that visits each node exactly once before returning to its start? Integer Linear Programming: Is there an integer vector x such that Ax>b? Page 15 Other complete problems L is called PSpace-hard (complete), iff for all L NP, L < p L (and L PSpace) Examples of PSpace complete problems: context-sensitive recognition (csr): Is a certain string w recognized by a context-sensitive grammar G? Csr is in NSpace(n) and DSpace(n 2 ) L is called P-hard (complete), iff for all L P, L < l L (and L P) Examples of P-complete problems: Emptyness of context-free grammars: Is the language defined by a context-free Grammar G empty? Circuit Value Problem CVP: Is a certain output y {0,1} m the correct result of a Boolean circuit with binary encoding α for binary input x 1,...,x n Examples of Nspace(log n) complete problems [using log-space reductions] Reachability in graphs: Is there a path from node x to y in a directed graph? Page 16
Content 1. Motivation: Example Algorithms 2. Computation models Register Machines, Turing Machine Church s Thesis Non-determinism 3. Main sequential complexity classes Uncomputable problems P time, NP time, P space, NP space P-time and log-space reductions NP complete problem examples P complete problems examples 4. Task Scheduling 1-processor, multi-processor, approximations Page 17 Task Scheduling: Problem formulation Input Set of Tasks T (finite or countable infinite) For each task t i T length : l(t i ) Deadline: d(t i ) Task precedence graph (acyclic) Number of processors to be used for scheduling: N N=1: single processor system N>1: multi-processor system Output Schedule σ, e.g. σ: T + (Task start time) σ: T (time-slotted schedule, slot-allocation) σ: x{1,...,n} T (multi-processor schedule) Goal: most tasks meet deadlines (to be specified better) Page 18
Task Scheduling: Problem formulation II Metrics Finish time of task set T Mean flow time Weighted mean flow time Fraction of tasks missing deadlines Etc. Many Variations, e.g. pre-emptive/non pre-emptive Communication costs/delays processor failures resource allocation probabilistic variations Page 19 Single processor scheduling: Example 1 Maximum constrained sequencing Given: Taskset T, length l(t i ), deadline d(t i ) Weight ( penalty ): w(t i ) Upper Bound on Penalty: K Goal: One-processor schedule σ: T + such that penalties of all tasks that miss deadlines does not exceed K: Σ F w(t i ) <= K, where F={t i σ(t i )+l(t i )>d(t i )} Discussion: branch-and-bound, A*, heuristics Page 20
Single processor scheduling: Example 2 Minimum precedence constrained sequencing with delays Given: Taskset T [,length l(t i )] Precedence graph: G=(T,E) Delays (with supremum D): 0<= d(t i ) <=D Goal: One-processor schedule σ: T + that Obeys precedence constraints Obeys delay constraints: < t i, t k > E, σ(t k )-σ(t i )> d(t i ) such that finish-time is minimized Discussion Page 21 Multi-processor scheduling: Example Minimum multi-processor scheduling Given: N processors Taskset T, length l(p,t i ), p=1,...,n Goal: N-processor schedule σ: T {1,...,N} such that finish-time is minimzed: max p {1,...,p} Σ σ -1 (p) l(p,t i ) Discussion Longest Processing Time (LPT) heuristics <= 4/3 of optimal schedule, even 4/3-1/(3N) for identical processors Minimized mean flow time: shortest processing time (SPT) first schedule optimal, O(n log n) for n tasks Page 22
Summary 1. Motivation: Example Algorithms 2. Computation models Register Machines, Turing Machine Church s Thesis Non-determinism 3. Main sequential complexity classes Uncomputable problems P time, NP time, P space, NP space P-time and log-space reductions NP complete problem examples P complete problems examples 4. Task Scheduling 1-processor, multi-processor, approximations Page 23 References Lecture notes: Computability and decidability, WS93/94, Prof. K-J Lange, TU Munich J. Hopcroft, J. Ullman: Introduction to automata theory, languages, and computation. Addison Wesley, 1979. Chapters 12 & 13 (Sects. 12.1-12.6, 13.1-13.2, 13.4-13.5) Garey, Johnson: Computers and Intractability: A Guide to the Theory of NP- Completeness, Freeman & Co, 1979 El-Rewini, Lewis, Ali: Task scheduling in parallel and distributed systems. Prentice Hall, 1994. Horowitz, Sahni: Exact and approximate algorithms for scheduling non-identical processors. Journal of the ACM, Vol. 23, No. 2, pp.317-327. 1976. http://www.nada.kth.se/~viggo/wwwcompendium/ Page 24