Turing & the Uncomputable; Intro to Time Complexity

Transcription

1 5-25: Great Ideas in Theoretical Computer Science Spring 209 Lecture 6 February 5, 209 Turing & the Uncomputable; Intro to Time Complexity Uncountable to uncomputable The real number /7 is computable. You could write a (non-halting) program (in your favorite language) which printed out all its digits: The same is true of 2, π, e, the first prime larger than 2 43,2,609, etc.; indeed, any real number you can think of. 2 Uncountable to uncomputable However, the set of all programs (in your favorite language) is just Σ *, for some finite alphabet Σ. Hence the set of all programs is countable. Computable Boolean functions An equivalence between languages and (Boolean-valued) functions: function f : {0,} * {0,} subset L {0,} * L = {x {0,} * : f(x) = } Hence the set of all computable reals is countable. But R is uncountable. Therefore there exist uncomputable reals. If L is decidable we call f computable, and vice versa. 3 4

2 Definition: Decidable languages A language L Σ * is decidable if there is a Turing Machine M which:. Halts on every input x Σ *. 2. Accepts inputs x L and rejects inputs x L. Such a Turing Machine is called a decider. It decides the language L. All languages? Decidable languages Regular languages EvenLength.. Factoring 0 n n Primality.. We like deciders. We don t like TM s that sometimes loop. 5 6 Encoding different objects with strings Fix some alphabet Σ. We use the notation to denote the encoding of an object as a string in Σ Examples: M Σ is the encoding a TM M D Σ is the encoding a DFA D M, M 2 Σ is the encoding of a pair of TMs M, M 2 M, x Σ is the encoding a pair M, x, where M is a TM, and x Σ is an input to M Question: Is every language in {0,} * decidable? Is every function f : {0,} * {0,} computable? Answer: No! Via a simple counting argument. Every TM is encodable by a finite string. Therefore the set of all TM s is countable. So the subset of all decider TM s is countable. Thus the set of all decidable languages is countable. But the set of all languages is uncountable. ( P({0,} * ) > {0,} * ) 7 8 2

3 Question: Is it just weirdo languages that no one would care about which are undecidable? Is it just weirdo languages that no one would care about which are undecidable? Answer (due to Turing, 936): Sadly, no. There are some very reasonable languages we d like to compute which are undecidable. 9 0 We need to write an autograder for isprime Kosbie s version Student submission returns True on exactly same inputs? True or False student submission isprime the correct program isprime Does such a program exist? i.e., can we solve/decide the following? Do they return True on exactly the same inputs? EQ = { M, M 2 M and M 2 are TMs s.t. L M = L(M 2 )} 2 3

4 Some undecidable languages Given two TM descriptions, M and M 2, do they act the same (accept/reject/loop) on all inputs? Given the description of an algorithm, M, does it print out HELLO WORLD? main(t,_,a ) char * a; { return! 0<t? t<3? main(-79,-3,a+ main(-87,-_, main(-86, 0, a+ ) +a)):, t<_? main( t+, _, a ) :3, main ( -94, -27+t, a ) &&t == 2?_ <3? main ( 2, _+, "%s %d %d\n" ) :9:6: t<0? t<-72? main( _, t, "@n'+,#'/*{}w+/w#cdnr/+,{}r/*de}+,/*{*+,/w{%+,/w#q#n+,/#{l,+,/n{n+,/+#n+,/#;#q#n+,/+k#;*+,/'r :'d*'3,}{w+k w'k:'+}e#';dq#'l q#'+d'k#!/+k#;q#'r}ekk#}w'r}ekk{nl]'/#;#q#n'){)#}w'){){nl]'/+#n';d}rw' i;# ){nl]!/n{n#'; r{#w'r nc{nl]'/#{l,+'k {rw' ik{;[{nl]'/w#q#n'wk nw' iwk{kk{nl]!/w{%'l##w#' i; :{nl]'/*{q#'ld;r'}{nlwb!/*de}'c ;;{nl'-{}rw]'/+,}##'*}#nc,',#nw]'/+kd'+e}+;#'rdq#w! nr'/ ') }+}{rl#'{n' ')# }'+}##(!!/") : t<-50? _==*a? putchar(3[a]): main(-65,_,a+) : main((*a == '/') + t, _, a + ) : 0<t? main ( 2, 2, "%s") :*a=='/' main(0, main(-6,*a, "!ek;dc i@bk'(q)- [w]*%n+r3#l,{}:\nuwloca-o;m.vpbks,fxntdceghiry"),a+);} This C program prints out all the lyrics of The Twelve Days Of Christmas. 3 4 Does the following program (written in Maple) print out HELLO WORLD? numbertotest := 2; flag := ; while flag = do flag := 0; numbertotest := numbertotest + 2; for p from 2 to numbertotest do if IsPrime(p) and IsPrime(numberToTest p) then flag := ; break; #exits the for loop end if end for end do print( HELLO WORLD ) It does so if and only if Goldbach s Conjecture is false. Some undecidable languages Given two TM descriptions, M and M 2, do they act the same (accept/reject/loop) on all inputs? Given the description of an algorithm, M, does it print out HELLO WORLD? Given a TM description M and an input x, does M halt on input x? Given a TM description M, does M halt when the input is a blank tape? 5 6 4

5 Some uncomputable functions This one is called The Halting Problem. Given a TM description M and an input x, does M halt on input x? Turing s Theorem: The Halting Problem is undecidable. Theorem: Proof: The Halting Problem is Undecidable Let HALTS Σ * be the language { M,x : M is a TM which halts on input x }. Then HALTS is undecidable. Assume for the sake of contradiction that M HALTS is a decider TM which decides HALTS. 7 8 The Halting Problem is Undecidable Here is the (high level) description of another TM called D, which uses M HALTS as a subroutine: D: Given as input M, the encoding of a TM M: D executes M HALTS ( M, M ). If this call accepts, D enters an infinite loop. If this call rejects, D halts (say, it accepts). The Halting Problem is Undecidable Assume M HALTS is a decider TM which decides HALTS. We can use it to construct a machine D such that D( M ) loops if M( M ) halts, halts if M( M ) loops. Time for the contradiction: Does D( D ) loop or halt? In other words D( M ) loops if M( M ) halts, halts if M( M ) loops. By definition, if it loops it halts and if it halts it loops. Contradiction

6 BTW: last part of proof basically the same as Cantor s Diagonal Argument. D( M ) loops if M( M ) halts, halts if M( M ) loops The set of all TM s is countable, so list them all: M M 2 M 3 M 4 M 5 BTW: last part of proof basically the same as Cantor s Diagonal Argument. D( M ) loops if M( M ) halts, halts if M( M ) loops The set of all TM s is countable, so list them all: M M 2 M 3 M 4 M 5 M halts halts loops halts loops M halts halts loops halts loops M 2 loops loops loops loops loops M 2 loops loops loops loops loops M 3 halts loops halts halts halts M 3 halts loops halts halts halts M 4 halts halts halts halts loops M 4 halts halts halts halts loops M 5 halts loops loops halts loops 2 22 M 5 halts loops loops halts loops D loops halts loops halts halts Given some code, determine if it terminates. It s not: we don t know how to solve it efficiently. It s not: we don t know if it s a solvable problem. In our proof that HALTS is undecidable, we used a hypothetical TM deciding HALTS to derive a contradiction Having established the undecidability of HALTS, we can show further problems to be undecidable using the powerful tool of REDUCTIONS We know that it is unsolvable by any algorithm

7 Reductions Using one problem as a subroutine to solve another problem. Informally, a reduction from A to B gives a way to solve problem A using a subroutine that can solve B Calculating the area of a rectangle reduces to calculating its length and height. Solving a linear system Ax = b reduces to computing the matrix inverse A Reductions Language A reduces to language B means (informally): there is a method that could be used to solve A if it has available to it a subroutine for solving B. The reduction gives such a method. Formally a (Turing) reduction from A to B is an oracle Turing machine that decides A when run with an oracle for B. Notation for A reduces to B: A T B (T stands for Turing). Think, A is no harder than B A is at least as easy as B Reducing language A to B Reductions Fact: Suppose A T B; i.e., A reduces to B. Input x y B? Yes/ No Decider TM with Oracle access to O Oracle O for B y N B? Yes/No Accept if x A Reject if x A If B is decidable, then A is also decidable. (can replace the assumed oracle for B with a decider for B, and the reduction can run this decider whenever it needs to ascertain membership of some string in B) Contrapositive: if A is undecidable then so is B. Think: B is at least as hard as A Reductions are the main technique for showing undecidability

8 Reductions examples Theorem: ACCEPTS = { M, x : M is a TM which accepts x} is undecidable. Proof: We ll prove HALTS reduces to ACCEPTS. Suppose O ACCEPTS is an oracle for language ACCEPTS. Then here s a description of an oracle TM deciding HALTS: Given M, x, run O ACCEPTS ( M, x ). If it accepts, then accept. Reverse the accept & reject states in M, forming M /. Run O ACCEPTS ( M /, x ). If it accepts (i.e., M rejects x), then accept. Else reject. Interesting observation To prove a negative result about computation (that a certain language is undecidable), you actual construct an algorithm namely, the reduction Theorem: Proof: Reductions another example EMPTY = { M : M accepts no strings} is undecidable. Let s prove ACCEPTS T EMPTY. This suffices, since we just showed ACCEPTS is undecidable. So suppose O EMPTY is an oracle for language EMPTY. Here s an oracle TM with oracle access to O EMPTY deciding ACCEPTS: Given M, x Write down the description N x of a TM N x which does the following: On input y, check if y=x. If not, reject. If so, simulate M on y. Then call upon the oracle O EMPTY on input N x and do the opposite. Correctness of reduction Code for N x : On input y, check if y=x. If not, reject. If so, simulate M on y. L(N x ) is either {x} or And L(N x ) = {x} precisely when M accepts x, i.e., M, x ACCEPTS Important: Reduction never runs N x ; it simply writes down the description N x of N x and probes the oracle whether N x EMPTY

9 Schematic of the reduction ACCEPTS T EMPTY Input M, x N x EMPTY? Yes/No Oracle TM deciding ACCEPTS Oracle O EMPTY. Write down description N x 2. Query oracle 3. Accept if O EMPTY answers No, and reject if it answers Yes Another example: ACCEPTS T INFINITE = Input M, x I x INFINITE? Oracle TM deciding ACCEPTS Oracle O INFINITE Yes/No. Write down description I x 2. Query oracle on I x 3. Accept if O INFINITE answers Yes, and reject if it answers No Note: Reduction is particularly simple: a single oracle query, and we just pass on answer to that query. Called mapping reduction Code I x : On input y: ignore y Run M on x & accept if it does. Remember: we don t run I x, we only write down it s code Undecidability galore Similar reductions can show undecidability of telling if, given an input TM M, L(M) is: Finite Regular Contains 25 in binary Decidable Contains a string of length more than 25 Contains only palindromes Etc etc Question: Is everything about TMs undecidable? Answer: No! Some problems about TMs behavior are decidable (as on your HW) Essentially any non-trivial property of languages

10 Question: Do all undecidable problems involve TM s? Post s Correspondence Problem Input: A finite collection of dominoes, having strings written on each half. Answer: No! Some very different problems are undecidable! E.g.: a ab a cabc bcc c Definition: A match is a sequence of dominoes, repetitions allowed, such that top string = bottom string Post s Correspondence Problem Input: A finite collection of dominoes, having strings written on each half. Post s Correspondence Problem Input: A finite collection of dominoes, having strings written on each half. E.g.: a ab a cabc bcc c Task: Output YES if and only if there is a match. Theorem (Post, 946): Undecidable. There is no algorithm solving this problem. Match: a ab bcc c a cabc bcc c = abccabcc = abccabcc (More formally, PCP = { Domino Set : there s a match} is an undecidable language.)

11 Post s Correspondence Problem Input: A finite collection of dominoes, having strings written on each half. Wang Tiles Input: Finite collection of Wang Tiles (squares) with colors on the edges. E.g., Task: Output YES if and only if there is a match. Theorem (Post, 946): Undecidable. There is no algorithm solving this problem. Two-second proof sketch: Given a TM M, you can make a domino set such that the only matches are execution traces of M which end in the accepting state. Hence ACCEPTS T PCP. Task: Output YES if and only if it s possible to make an infinite grid from copies of them, where touching sides must color-match. Theorem (Berger, 966): Undecidable Mortal Matrices Input: Two 5 5 matrices of integers, A & B. Question: Is it possible to multiply A and B together (multiple times in any order) to get the 0 matrix? Hilbert s 0 th problem Input: Multivariate polynomial w/ integer coeffs. Question: Does it have an integer root? Theorem (970): Undecidable. Theorem (Cassaigne, Halava, Harju, Nicolas, 204): Undecidable. Matiyasevich Robinson Davis Putnam 44 45

12 Hilbert s 0 th problem Input: Multivariate polynomial w/ integer coeffs. Question: Does it have an integer root? Undecidable. Entscheidungsproblem Input: A sentence in first-order logic. n, x, y, z N: n 3 (x n + y n = z n ) Question: Is it provable? Question: Does it have a real root? Decidable. Tarski, 95. This is undecidable. We ll come back to this in the lecture on Gödel s Incompleteness Theorem Question: Does it have a rational root? Not known if it s decidable or not Introduction to Time Complexity What have we done so far? What will we do next?

13 What have we done so far? > Introduction to the course Computer science is no more about computers than astronomy is about telescopes. > Strings and Encodings What have we done so far? > The study of computation Computability/Decidability - Most problems are undecidable. - Some very interesting problems are undecidable. > Formalization of computation/algorithm Deterministic Finite Automata Turing Machines But many interesting problems are decidable! 50 5 What is next? > The study of computation Computability/Decidability Computational Complexity (Practical Computability) - How do we define computational complexity? - What is the right level of abstraction to use? - How do we analyze complexity? - What are some interesting problems to study? - What can we do to better understand the complexity of problems? Why is computational complexity important? complexity ~ practical computability Simulations (e.g. of physical or biological systems) - tremendous applications in science, engineering, medicine, Optimization problems - arise in essentially every industry Social good - finding efficient ways of helping others Artificial intelligence list goes on Security, privacy, cryptography.. - applications of computationally hard problems

14 Why is computational complexity important? Goals for next 2 lectures. What is the right way to study complexity? million dollar question (or maybe 6 million dollar question) - using the right language and level of abstraction - upper bounds vs lower bounds - polynomial time vs exponential time 2. Appreciating the power of algorithms. - analyzing some cool (recursive) algorithms Guiding principles What is the right language and level of abstraction for studying computational complexity? Size matters Value matters What is the meaning of: The (asymptotic) complexity of algorithm A is O n 2. Model matters

15 Size matters sorting bazillion numbers > sorting 2 numbers. GREAT IDEA # Running time of an algorithm depends on input length. n = input length/size Running time of an algorithm is a function of n. n is usually: sometimes: # bits in a binary encoding of input. explicitly defined to be something else. (But what is n going to be mapped to?) We have to be careful Value matters Not all inputs are created equal! Size matters Value matters Among all inputs of length n: - some might take 2 steps - some might take bazillion steps. Model matters

16 Why worst-case? GREAT IDEA # 2 Running time of an algorithm is a worst-case function of n. n # steps taken by the worst input of length n We are not dogmatic about it. Can study average-case (random inputs) Can try to look at typical instances. Can do smoothed analysis. BUT worst-case analysis has its advantages: - An ironclad guarantee. - Hard to define typical instances. - Random instances are often not representative. - Often much easier to analyze We have to be careful Model matters PAL = { x 0, x = x R } Size matters Value matters Model matters How many steps required to decide PAL? Facts: O n 2 is the best for -tape TMs. O(n) is the best for 2-tape TMs. Alert: Just because the obvious/standard way to solve it on single-tape TMs takes O n 2 time does NOT mean there isn t a better way Algorithms can be clever and unexpected. Lower bounds are a difficult enterprise

17 Model matters Model matters How many steps required to decide L? Facts: is the best for -tape TMs. is the best for 2-tape TMs. A function in Python: def twofingers(s): lo = 0 hi = len(s)- while (lo < hi): if (s[lo]!= 0 or s[hi]!= ): return False lo += hi -= return True # of steps 3? 4? 5? Seems like Model matters Model matters hi -= Initially hi = n- How many bits to store hi? If n- is a power of 2: hi = hi = 0 steps A function in Python: def twofingers(s): lo = 0 hi = len(s)- while (lo < hi): if (s[lo]!= 0 or s[hi]!= ): return False lo += hi -= return True # of steps 3? 4? 5? log n??

18 Model matters Model matters Initially lo = 0, hi = n- if (s[lo]!= 0 or s[hi]!= ): Does it take n steps to go from s[0] to s[n-]? A function in Python: def twofingers(s): lo = 0 hi = len(s)- while (lo < hi): if (s[lo]!= 0 or s[hi]!= ): return False lo += hi -= return True # of steps n?? log n?? Choice of computational model GREAT IDEA # 3 Computational model does matter for running time. Which model is the best model? No such thing. Any reasonable model should be able to simulate another reasonable model with only a polynomial increase in running time. - For example, -tape TM can simulate 2-tape TM with quadratic slowdown

19 Which model does this correspond to? GREAT IDEA # 4 All reasonable deterministic models are polynomially equivalent. def twofingers(s): lo = 0 hi = len(s)- while (lo < hi): if (s[lo]!= 0 or s[hi]!= ): return False lo += hi -= return True 3? 4? 5? 80 8 The Random-Access Machine (RAM) model Good combination of reality/simplicity. +, -, /, *, <, >, etc. e.g. 245*2894 takes step memory access Actually: e.g. A[94] Assume arithmetic operations take step IF numbers are bounded by poly(n). takes step Putting great ideas #, # 2 and # 3 together Unless specified otherwise, we use this model. (more on this next lecture)

20 Defining running time With a specific computational model in mind: Definition: The running time of an algorithm A is a function defined by worst-case Need one more level of abstraction There is a TM that decides PALINDROME in time Analogous to too many significant digits. Wow, TMI Write when A is clear from context Need one more level of abstraction There is a TM that decides PALINDROME in time Palindrome TM has running time T n = 2 n n + We want to use the right level of abstraction! Analogous to too many significant digits. For algorithms at a higher level (like, in C, or pseudocode), it s not exactly clear what counts as time step Even for slightly more complicated algorithms, it s nearly impossible to calculate so precisely The key takeaway of this T(n): it s quadratic ; that is, proportional to n 2. This leads us to

21 Great Idea #5: Big-O notation The CS way to compare functions: Big O Our notation for when comparing functions. The right level of abstraction! Sweet spot - coarse enough to suppress details like programming language, compiler, architecture, - sharp enough to make comparisons between different algorithmic approaches Big O Big O Informal: An upper bound that ignores constant factors and ignores small n. For roughly means up to a constant factor and ignoring small n

22 Big O Big O For roughly means up to a constant factor and ignoring small n. Formal Definition: For, we say if there exist constants, such that for all, we have. ( and cannot depend on. ) Formal Definition: For, we say if there exist constants, such that for all, we have. ( and cannot depend on. ) Big O

23 Poll Same example: Take. When Select all that apply. is: Beats me Big O practice Run time scaling Running-time: double the input Ratio: constant double the input double the input Note on notation: People usually write Another valid notation: 0 02 double the input double the input (constant) 23

24 Big O Big O Common Big O classes and their names Constant: Logarithmic: Square-root: Linear: Quasi-linear: Quadratic: Polynomial: Exponential: (for some constant k > 0) (for some constant k > 0) n vs 2 n 2 n n ,048,576 20,073,74,824 30,52,92,504,606,846, Exponential running time If your algorithm has exponential running time e.g. Then it is (usually) not practical

25 Exponential running time: Example Given a list of integers, determine if there is a subset of the integers that sum to Exhaustive Search (Brute Force Search): Try every possible subset and see if it sums to 0. Number of subsets is 2 n So running time is at least 2 n 07 25