Algorithms and Architectures III: Distributed Systems H-P Schwefel, Jens M. Pedersen Mm6 Distributed storage and access (jmp) Mm7 Introduction to security aspects (hps) Mm8 Parallel complexity (hps) Mm9 Load Balancing (JMP) Mm10 tbd hps@kom.auc.dk http://www.kom.auc.dk/~hps Page 1 Complexity: Some examples Graph Theory Finding shortest paths between all node pairs in a graph: Floyd Algorithm, time O(N^3), space O(N^2), N= Number of Nodes Find shortest paths from specific node to all destinations: Dijkstra Algorithm, time O(N^2) Search & Optimization Problems: Binary search in ordered set: O(log(N)) Ordering set of N elements: O(N log (N)) Traveling Salesman problem (shortest tour between N cities): Exhaustive Search, time O(N!) Finding an allocation of the N boolean variables x 1,...,x N such that the boolean function f(x 1,...,X N )=true: exhaustive search, time O(2^N) Finding the prime-factors of a large composite n=p*q with p,q=prime Interesting Questions: Can these (and other) problems be solved faster? Are there problems that cannot be computed by an algorithm at all? Is true parallelism in distributed systems helpful? Page 2
Content 1. Motivation: Example Algorithms 2. Sequential computation models Register Machines, Turing Machine Church s Thesis Non-determinism 3. Main sequential complexity classes Uncomputable problems P time, NP time, P space, NP space P-time and log-space reductions NP complete problem examples P complete problems examples 4. Parallel Computation Models and Classes PRAM, Circuits, alternating Turing Machine Parallel complexity classes 5. Summary/Exercises Page 3 Computational Models & Computability Finite program (normally represented as string over some alphabet) Finite number of registers Assignments and basic arithmetics: Ri=0, Ri=Rj+Rk, INC/DEC Ri Different types of repetitive instructions Loop Ri While Ri 0 If Ri 0 goto label Recursion Tail recursion/primitive recursion µ-recursion/partial recursion Loop computable functions = Tail-recursive functions While-computable functions = if-goto computable functions = partial recursive functions Page 4
Computational Models: Turing Machine Informal Definition Turing Machine = automaton with finite state-set working on an infinite storage organized as read/write tape Operation: read one symbol from the current position on the tape; depending on read symbol and current state determine New state New symbol to be written at the current tape position Movement of current position to right or left For certain states and certain input symbols, the Turing machine may stop operation Turing Machine <Q,Γ,b,X, q 0,F,δ> Finite State-space Q Finite working alphabet Γ Empty symbol b Finite input Alphabet X Γ/{b} Starting state q 0 Q F Q is set of accepting final states Transition function δ: (Q/F) x Γ Γ x {L,R} x Q Page 5 Turing Machines & Computability Turing machines (cntd) Example: Addition of unary numbers Equally power-ful models (with respect to computability) Only one-sided infinite tape Multiple tapes Multiple read/write heads Multiple dimensions Turing computable = if-goto computable (extended on X* = set of sequences over X) Church Thesis: Any algorithmic computable function is Turing computable Page 6
Acceptance, decidability; diagonalisation L X* is called Turing acceptable (recursive enumerable), iff there is a TM M that Stops on all inputs v L in an acceptance state q f F For all v L, M may or may not stop, but if it stops, then in a state F L X* is called decidable (recursive), if there is a TM M that Stops on all inputs v X* but only accepts if v L Corollaries: L decidable X*/L decidable L acceptable and X*/L acceptable L decidable Diagonalisation: Let <M> be a binary encoding of the Turing Machine M Hence, all possible TMs can be enumerated w i =<M i > L d := { w i w i L(M i )}, there does not exist any TM that accepts L d Page 7 Non-determinism Non-deterministic Turing Machine M nd =<Q,Γ,b,X, q 0,F,τ> Finite State-space Q Finite working alphabet Γ Empty symbol b Finite input Alphabet X Γ/{b} Starting state q 0 Q F Q is set of accepting final states Transition function τ: (Q/F) x Γ (Γ x {L,R} x Q) Consequence: At each computation step, multiple (but finite) number of choices v X* accepted by M nd iff there exists one computation in the computation tree that accepts v L X* acceptable by a deterministic TM L acceptable by a non-determinstic TM Page 8
Content 1. Motivation: Example Algorithms 2. Sequential computation models Register Machines, Turing Machine Church s Thesis Non-determinism 3. Main sequential complexity classes Uncomputable problems P time, NP time, P space, NP space P-time and log-space reductions NP complete problem examples P complete problems examples 4. Parallel Computation Models and Classes PRAM, Circuits, alternating Turing Machine Parallel complexity classes 5. Summary/Exercises Page 9 Un-computable Problems Examples: L NE :={ w i L(M i ) } is Turing acceptable (recursive enumerable) L E :={ w i L(M i ) = } is not Turing decidable (recursive), hence also not Turing acceptable L h :={<M,w> M stops on input w}; L nh :=X*/L h ={<M,w> M does not stop on input w} L h is Turing acceptable but not decidable, L nh is not Turing acceptable It is not decidable whether a context-free grammar creates a regular language It is not decidable whether two context-free grammars have a non-empty intersection Page 10
Resources: Time and Space complexity Computational Model: Deterministic Turing Machine M with k one-sided infinite tapes Read-only tape that contains input of length n If for all inputs of length n M uses at most S(n) fields on any of the k tapes M has space complexity S(n) M uses at most T(n) steps M has time complexity T(n) Example: L={wbw R w {0,1}*} (Palindromes) Space complexity S(n)=log2(n) (using a binary counter) Time complexity T(n)=n+1 (while using space (n-1)/2) Page 11 Non-deterministic classes and relations L NSpace(S(n)): Iff there exists a non-deterministic TM that decides membership of L with no computation sequence exceeding S(n) fields on any tape L NTime(T(n)): Iff there exists a non-determinstic TM that decides membership of L with no computation requireing more than T(n) steps Similar Complexity classes for deterministic case: Dspace(S(n)), Ntime(T(n)) Relations (by definition): DTime(T(n)) NTime(T(n)) DTime(f(n)) DSpace(f(n)), NTime(f(n)) NSpace(f(n)) DSpace(S(n)) NSpace(S(n)) Reverse Directions: L DSpace(f(n)) and f(n)>log 2 (n) there exists a c>0 with L DTime(c f(n) ) L NTime(T(n)) there exists a c>0 with L DTime(c T(n) ) Page 12
Space compression and linear speed-up Due to freedom of choice of state-space and working alphabet (Notation: x=n or x=d, but consistently replaced within one line) Linear factors in complexity classes L xspace(s(n)) L xspace(c*s(n)), for all c>0 L xtime(t(n)), and lim T T(n)/n= L xtime(c*t(n)) for all c>0 L xtime(c*n) L xtime((1+ε)*n) for all ε>0 Reduction from k-tape TM L xspace(s(n)) for k tapes L xspace(s(n)) for TM with one tape L xtime(t(n)) for k>1 tapes L xtime(t 2 (n)) for single tape TM L xtime(t(n)) for k>2 tapes L xtime(t(n) log T(n)) for TM with 2 tapes Page 13 Polynomial time and space: P, NP, PSpace Time Efficient Problems: Polynomial Time P P := PTime := ² i 1 DTime(n i ) Non-deterministic polynomial Time: NP NP := NPtime := ² i 1 NTime(n i ) Polynomial Space: Pspace PSpace := ² i 1 DSpace(n i ) Relation: DSpace(log n) P NP PSpace And it is known: DSpace(log n) PSpace, hence at least one of the relations is a true subset in the chain above, but it is not known which one(s) Page 14
Reductions, hard and complete problems Definition: L < g L, L is reducible to L via the function g iff x L g(x) L For complexity investigations: resource limited reductions Polynomial time reductions < p : g(x) computable in deterministic polynomial time Log-Space reductions < l : g(x) computable using deterministic logarithmic space L is called NP-hard, iff for all L NP, L < p L L is called NP-complete, iff L NP-hard and L NP Examples of NP complete problems: SAT: Do the variables X 1,...,X n in a boolean expression have an allocation such that the expression=true? (Directed) Hamilton Circles: Is there a path in the (directed) graph G that visits each node exactly once before returning to its start? Integer Linear Programming: Is there an integer vector x such that Ax>b? Page 15 Other complete problems L is called PSpace-hard (complete), iff for all L NP, L < p L (and L PSpace) Examples of PSpace complete problems: context-sensitive recognition (csr): Is a certain string w recognized by a context-sensitive grammar G? Csr is in NSpace(n) and DSpace(n 2 ) L is called P-hard (complete), iff for all L NP, L < l L (and L P) Examples of P-complete problems: Emptyness of context-free grammars: Is the language defined by a context-free Grammar G empty? Circuit Value Problem CVP: Is a certain out y {0,1} m the correct result of a Boolean circuit with binary encoding α for binary input x 1,...,x n Examples of Nspace(log n) complete problems [using log-space reductions] Reachability in graphs: Is there a path from node x to y in a directed graph? Page 16
Content 1. Motivation: Example Algorithms 2. Sequential computation models Register Machines, Turing Machine Church s Thesis Non-determinism 3. Main sequential complexity classes Uncomputable problems P time, NP time, P space, NP space P-time and log-space reductions NP complete problem examples P complete problems examples 4. Parallel Computation Models and Classes PRAM, Circuits, [alternating Turing Machine] Parallel complexity classes 5. Summary/Exercises Page 17 Parallel Computation the big picture Fundamental Limitation for execution on P processors parallel time best sequential time / number of processors Roughly speaking, we will see in the following: Parallel Time sequential Space Sequential Time parallel Hardware Page 18
Parallel Computation Models I: Circuits Uniform Boolean Circuit families Boolean Circuit α = directed acyclic graph, labeled vertices express boolean functions Calculate functions f: {0,1} n {0,1} m Size(α) = number of vertices, Depth(α)=length of longest path Binary encoding of α as string α of length O(size(α) log(size(α)) ) Logarithmic space uniform circuit families α n can be computed from input 1 n by a deterministic TM in space O(log(size(α n ))) Uniform circuit families with binary output define accepted language L: w L f α w (w) = 1 Page 19 Parallel Computation Models II: PRAMs Parallel Random Access Machine (PRAM) Random Access Machine (RAM) Computation Unit with fixed user-defined program Read-only input tape, write-only output tape Unbounded number of memory cells Basic instructions: arithmetic, assignments, control operations (if-goto) Time=number of instructions executed, Space=number of memory cells used PRAM Extensions Unbounded collection of numbered RAM processors: P1,P2,... Each with local memory, synchronized execution of instructions Unbounded collection of shared memory cells (containing input and output): C1,C2,... Instructions for direct and indirect read/write access to shared memory Concurrent Read and concurrent write: CRCW PRAM Concurrent read and exclusive write: CREW PRAM Exclusive read and exclusive write: EREW PRAM Etc. Commands for activating certain number of processors by P0 f:{0,1}* {0,1}* computable in parallel time t(n) and processors p(n), iff there exists PRAM M that on input x outputs f(x) in time t(n) and using p(n) processors Page 20
Computation Models: Equivalences f: {0,1}* {0,1}* computable by logarithmic space uniform Boolean circuit family (α n ) with depth log(n) k and polynomial size f can be computed by CREW PRAM M in time log(n) j on polynomial number of processors NC k :=set of all languages L recognized by uniform Boolean circuit family with Polynomial size and depth=o(log(n) k ) NC := ² k 1 NC k Theorem: If any P-complete problem is in NC, then NC=P. However there is strong belief that NC P. If this is true, then P-complete problems are not efficiently parallelizable! s(n) log(n) DSpace(s(n)) CRCW-Time-Proc(s(n), 2 O(s(n)) ) Page 21 Summary 1. Motivation: Example Algorithms 2. Sequential computation models Register Machines, Turing Machine Church s Thesis Non-determinism 3. Main sequential complexity classes Uncomputable problems P time, NP time, P space, NP space P-time and log-space reductions NP complete problem examples P complete problems examples 4. Parallel Computation Models and Classes PRAM, Circuits, [alternating Turing Machine] Parallel complexity classes 5. Summary Page 22
References/Acknowledgments Lecture notes: Computability and decidability, WS93/94, Prof. K-J Lange, TU Munich J. Hopcroft, J. Ullman: Introduction to automata theory, languages, and computation. Addison Wesley, 1979. Chapters 12 & 13 (Sects. 12.1-12.6, 13.1-13.2, 13.4-13.5) R. Greenlaw, H. J. Hoover, W. Ruzzo: Limits to Parallel Computation P- Completeness Theory. Oxford University Press, 1995. Chapters 1-4 Page 23