Lecture 13: Foundations of Math and Kolmogorov Complexity

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Lecture 13: Foundations of Math and Kolmogorov Complexity"

Transcription

1 6.045 Lecture 13: Foundations of Math and Kolmogorov Complexity 1

2 Self-Reference and the Recursion Theorem 2

3 Lemma: There is a computable function q : Σ* Σ* such that for every string w, q(w) is the description of a TM P w that on every input, prints out w and then accepts Proof Define a TM Q: s w Q P w w 3

4 Theorem: There is a Self-Printing TM Proof: First define a TM B which does this: M B w P M M M Now consider the TM that looks like this: w B w P B B P B B B No explicit self-reference here! QED 4

5 The Recursion Theorem Theorem: For every TM T computing a function t : Σ* Σ* Σ* there is a Turing machine R computing a function r : Σ* Σ*, such that for every string w, r(w) = t(r, w) (a,b) T t(a,b) w R t(r,w) 5

6 Proof: (a,b) T t(a,b) Define M = N B N w B Y w T w P N N w N Define R: What is S? w P M M B S w T t(s,w) 6

7 Proof: (a,b) Define M = T M t(a,b) B What is M(M,w)? N w B Y w T w P M M w M Define R: S M S w P B T t(s,w) M w 7

8 Proof: (a,b) T t(a,b) M B w M PM w B Y w T Define R: S S = Y = R. QED M S w P B T t(s,w) M w 8

9 FOO x (y) := Output x and halt. BAR(M) := Output N(w) = Run FOO M outputting M. Run M on (M, w) Q(N, w) := Run BAR(N) outputting S. Run T on (S, w) R(w) := Run FOO Q outputting Q. Run BAR(Q) outputting S. Run T on (S, x) Claim: S is a description of R itself! S(w) = Run FOO Q outputting Q. Run Q on (Q, w)

10 FOO x (y) := Output x and halt. BAR(M) := Output N(w) = Run FOO M outputting M. Run M on (M, w) Q(N, w) := Run BAR(N) outputting S. Run T on (S, w) R(w) := Run FOO Q outputting Q. Run BAR(Q) outputting S. Run T on (S, w) Claim: S is a description of R itself! S(w) = Run FOO Q outputting Q. Run BAR(Q) outputting S. Run T on (S, w) Therefore R(w) = T(R, w)

11 For every computable t, there is a computable r such that r(w) = t(r,w) where R is a description of r Suppose we can design a TM T of the form: On input (x,w), do bla bla with x, do bla bla bla with w, etc. etc. We can then find a TM R with the behavior: On input w, do bla bla with R, do bla bla bla with w, etc. etc. We can use the operation: Obtain your own description in Turing machine pseudocode! 11

12 Theorem: A TM is undecidable Proof (using the recursion theorem) Assume H decides A TM Construct machine B such that on input w: 1. Obtains its own description B 2. Runs H on (B, w) and flips the output Running B on input w always does the opposite of what H says it should! A formalization of free will paradoxes! No single machine can predict behavior of all others 12

13 Turing Machine Minimization MIN = {M M is a minimal-state Turing machine} Theorem: MIN is unrecognizable! Proof: Suppose we could recognize MIN with TM M M(x) := Obtain description of M. For k = 1,2,, Run M on the TMs M 1, M k for k steps, until M accepts some M i with Q(M i ) > Q(M). Output M i (x). We have: 1. L M = L M i [by construction] 2. M has fewer states than M i 3. M i is minimal [by definition of MIN] CONTRADICTION! 13

14 Computability and the Foundations of Mathematics 17

15 Formal Systems of Mathematics A formal system describes a formal language for - writing (finite) mathematical statements, - has a definition of a proof of a statement Example: Every TM M defines some formal system F - {Mathematical statements in F } = * String w represents the statement M halts on w - A proof that M halts on w can be defined to be the computation history of M on w: the sequence of configurations C 0 C 1 C t that M goes through while computing on w Could sometimes prove M doesn t halt on w 18

16 Interesting Systems of Mathematics Define a formal system F to be interesting if: 1. Any mathematical statement about computation can be (computably) described as a statement of F. Given (M, w), there is a (computable) S M,w of F such that S M,w is true in F if and only if M accepts w. 2. Proofs are convincing a TM can check that a candidate proof of a theorem is correct This set is decidable: {(S, P) P is a proof of S in F } 3. If S is in F and there is a proof of S describable as a computation, then there s a proof of S in F. If M halts on w, then there s either a proof P of S M,w or a proof P of S M,w 19

17 Consistency and Completeness A formal system F is inconsistent if there is a statement S in F such that both S and S are provable in F F is consistent if it is NOT inconsistent A formal system F is incomplete if there is a statement S in F such that neither S nor S are provable in F F is complete if it is NOT incomplete 20

18 Limitations on Mathematics! For every consistent and interesting F, Theorem 1. (Gödel 1931) F must be incomplete! There are mathematical statements that are true but cannot be proved. Theorem 2. (Gödel 1931) The consistency of F cannot be proved in F. Theorem 3. (Church-Turing 1936) The problem of checking whether a given statement in F has a proof is undecidable. 21

19 Unprovable Truths in Mathematics (Gödel) Every consistent interesting F is incomplete: there are statements that cannot be proved or disproved. Let S M, w in F be true if and only if M accepts w Proof: Define TM G(w): 1. Obtain own description G [Recursion Theorem!] 2. For all strings P in lexicographical order, If (P is a proof of S G, w in F ) then reject If (P is a proof of S G, w in F ) then accept Note: If F is complete then G cannot run forever! 1. If (G accepts w) then have proof P of G doesn t accept w 2. If (G rejects w) then found proof P of G accepts w In either case, F is inconsistent! Proof of S G, w and S G, w 22

20 (Gödel 1931) The consistency of F cannot be proved within any interesting consistent F Proof: Assume we can prove F is consistent in F We constructed :S G, w = G does not accept w which is true, but has no proof in F G does not accept w There is no proof of :S G, w in F But if there s a proof of F is consistent in F, then there is a proof of :S G, w in F (here s the proof): If S G, w is true, then there is a proof in F of S G, w and a proof in in F of :S G, w But since F is consistent, this cannot be true. Therefore, :S G, w is true This contradicts the previous theorem! 23

21 Undecidability in Mathematics PROVABLE F = {S there s a proof in F of S, or there s a proof in F of :S} (Church-Turing 1936) For every interesting consistent F, PROVABLE F is undecidable Proof: Suppose PROVABLE F is decidable with TM P. Then we can decide A TM with the following procedure: On input (M, w), run the TM P on input S M,w If P accepts, examine all proofs in lex order If a proof of S M,w is found then accept If a proof of :S M,w is found then reject If P rejects, then reject. Why does this work? 24

22 Kolmogorov Complexity: A Universal Theory of Data Compression 25

23 The Church-Turing Thesis: Everyone s Intuitive Notion of Algorithms = Turing Machines This is not a theorem it is a falsifiable scientific hypothesis. A Universal Theory of Computation 26

24 A Universal Theory of Information? Can we quantify how much information is contained in a string? A = B = Idea: The more we can compress a string, the less information it contains. 27

25 Information as Description Thesis: The amount of information in a string x is the length of the shortest description of x Let x 2 {0,1}* How should we describe strings? Use Turing machines with inputs! Def: A description of x is a string <M,w> such that M on input w halts with only x on its tape. Def: The shortest description of x, denoted as d(x), is the lexicographically shortest string <M,w> such that M(w) halts with only x on its tape. 28

26 A Specific Pairing Function Theorem. There is a 1-1 computable function <,> : Σ* x Σ* Σ* and computable functions 1 and 2 : Σ* Σ* such that: z = <M,w> iff 1 (z) = M and 2 (z) = w Define: <M,w> := 0 M 1 M w (Example: <10110,101> = ) Note that <M,w> = 2 M + w

27 Kolmogorov Complexity (1960 s) Definition: The shortest description of x, denoted as d(x), is the lexicographically shortest string <M,w> such that M(w) halts with only x on its tape. Definition: The Kolmogorov complexity of x, denoted as K(x), is d(x). EXAMPLES?? Let s first determine some properties of K. Examples will fall out of this. 30

28 A Simple Upper Bound Theorem: There is a fixed c so that for all x in {0,1}* K(x) x + c The amount of information in x isn t much more than x Proof: Define a TM M = On input w, halt. On any string x, M(x) halts with x on its tape. Observe that <M,x> is a description of x. Let c = 2 M +1 Then K(x) <M,x> 2 M + x + 1 x + c 31

29 Repetitive Strings have Low K-Complexity Theorem: There is a fixed c so that for all n 2, and all x ϵ {0,1}*, K(x n ) K(x) + c log n The information in x n isn t much more than that in x Proof: Define the TM N = On input <n,<m,w>>, Let x = M(w). Print x for n times. Let <M,w> be the shortest description of x. Then K(x n ) K(<N,<n,<M,w>>>) 2 N + d log n + K(x) c log n + K(x) for some constants c and d 32

30 Repetitive Strings have Low K-Complexity Theorem: There is a fixed c so that for all n 2, and all x ϵ {0,1}*, K(x n ) K(x) + c log n The information in x n isn t much more than that in x Recall: A = For w = (01) n, we have K(w) K(01) + c log n So for all n, K((01) n ) d + c log n for a fixed c, d 33

31 Does The Computational Model Matter? Turing machines are one programming language. If we use other programming languages, could we get significantly shorter descriptions? An interpreter is a semi-computable function p : Σ* Σ* Takes programs as input, and (may) print their outputs Definition: Let x ϵ {0,1}*. The shortest description of x under p, called d p (x), is the lexicographically shortest string w for which p(w) = x. Definition: The K p complexity of x is K p (x) := d p (x). 34

32 Does The Computational Model Matter? Theorem: For every interpreter p, there is a fixed c so that for all x ϵ {0,1}*, K(x) K p (x) + c Moral: Using another programming language would only change K(x) by some additive constant Proof: Define M = On w, simulate p(w) and write its output to tape Then <M,d p (x)> is a description of x, so K(x) <M,d p (x)> 2 M + K p (x) + 1 c + K p (x) 35

33 There Exist Incompressible Strings Theorem: For all n, there is an x ϵ {0,1} n such that K(x) n There are incompressible strings of every length Proof: (Number of binary strings of length n) = 2 n but (Number of descriptions of length < n) (Number of binary strings of length < n) = n-1 = 2 n 1 Therefore, there is at least one n-bit string x that does not have a description of length < n 36

34 Random Strings Are Incompressible! Theorem: For all n and c 1, Pr x ϵ {0,1} n[ K(x) n-c ] 1 1/2 c Most strings are highly incompressible Proof: (Number of binary strings of length n) = 2 n but (Number of descriptions of length < n-c) (Number of binary strings of length < n-c) = 2 n-c 1 Hence the probability that a random x satisfies K(x) < n-c is at most (2 n-c 1)/2 n < 1/2 c. 37

35 KOLMOGOROV DIRECTIONS 38

36 Kolmogorov Complexity: Try it! Give short algorithms for generating the strings:

37 Kolmogorov Complexity: Try it! Give short algorithms for generating the strings:

38 Kolmogorov Complexity: Try it! Give short algorithms for generating the strings:

39 Kolmogorov Complexity: Try it! Give short algorithms for generating the strings: This seems hard to determine in general. Why? 42

40 Computing Compressibility? Can an algorithm perform optimal compression? Can algorithms tell us if a given string is compressible? COMPRESS = { (x,c) K(x) c} Theorem: COMPRESS is undecidable! Idea: If decidable, we could design an algorithm that prints the shortest incompressible string of length n But such a string could then be succinctly described, by providing the algorithm code and n in binary! Berry Paradox: The smallest integer that cannot be defined in less than thirteen words. 43

41 Computing Compressibility? COMPRESS = {(x,c) K(x) c} Theorem: COMPRESS is undecidable! Proof: Suppose it s decidable. Consider the TM: M = On input x ϵ {0,1}*, let N = 2 x. For all y ϵ {0,1}* in lexicographical order, If (y,n) COMPRESS then print y and halt. M(x) prints the shortest string y with K(y ) > 2 x. <M,x> is a description of y, and <M,x> d + x So 2 x < K(y ) d + x. CONTRADICTION for large x! 44

42 Yet Another Proof that A TM is Undecidable! COMPRESS = {(x,c) K(x) c} Theorem: A TM is undecidable. Proof: Reduction from COMPRESS to A TM. Given a pair (x,c), our reduction constructs a TM: M x,c = On input w, For all pairs <M,w > with <M,w > c, simulate each M on w in parallel. If some M halts and prints x, then accept. K(x) c M x,c accepts ε 45

43 More on Interesting Formal Systems A formal system F is interesting if it is finite and: 1. Any mathematical statement about computation can also be effectively described within F. For all strings x and integers c, there is a S x,c in F that is equivalent to K(x) c 2. Proofs are convincing: it should be possible to check that a proof of a theorem is correct This set is decidable: { (S,P) P is a proof of S in F } 46

44 The Unprovable Truth About K-Complexity Theorem: For every interesting consistent F, There is a t s.t. for all x, K(x) > t is unprovable in F Proof: Define an M that treats its input as an integer: M(k) := Search over all strings x and proofs P for a proof P in F that K(x) > k. Output x if found Suppose M(k) halts. It must print some output x Then K(x ) = K(<M,k>) c + k c + log k for some c Because F is consistent, K(x ) > k is true But k < c + log k only holds for small enough k If we choose t to be greater than these k then M(t) cannot halt, so K(x) > t has no proof! 47

45 Random Unprovable Truths Theorem: For every interesting consistent F, There is a t s.t. for all x, K(x) > t is unprovable in F For a randomly chosen x of length t+100, K(x) > t is true with probability at least 1-1/2 100 We can randomly generate true statements in F which have no proof in F, with high probability! For every interesting formal system F there is always some finite integer (say, t=10000) so that you ll never be able to prove in F that a random bit string requires a bit program! 48

46 Proving Theorems With K-Complexity Theorem: L = {x x x 0, 1 } is not regular. Proof: Suppose L is recognized by a DFA D. Let n 0 and choose an x 0, 1 such that K x n Let q x be the state of D reached after reading in x Define a TM M(D, q, n): Find a path P in D of length n that starts from state q and ends in an accept state. Print the n-bit string along path P, and halt. Claim: The pair <M,(D, q x, n)> is a description of x! So n K x <M,(D, q x, n)> O log n CONTRADICTION! 50

47 Next Episode: Complexity Theory! 51

48 Repetitive Strings have Low Information Theorem: There is a fixed c so that for all x ϵ {0,1}* K(xx) K(x) + c The information in xx isn t much more than that in x Proof: Let N = On <M,w>, let s = M(w). Print ss. Suppose <M,w> is the shortest description of x. Then <N,<M,w>> is a description of xx Therefore K(xx) <N,<M,w>> 2 N + <M,w> N + K(x) + 1 c + K(x) 57

49 Information as Description Thesis: The amount of information in a string x is the length of the smallest description of x Let x 2 {0,1}* How should we describe strings? Use Turing Machines! Def. The shortest description of x, denoted d(x), is the lexicographically shortest string M such that the TM M on input ε halts with only x on its tape. 58

50 Kolmogorov Complexity (1960 s) Def. The shortest description of x, denoted d(x), is the lexicographically shortest string M such that the TM M on input ε halts with only x on its tape. Definition: The Kolmogorov complexity of x, denoted as K(x), is d(x). EXAMPLES?? Let s first determine some properties of K. Examples will fall out of this. 59

51 A Simple Upper Bound Theorem: There is a fixed c so that for all x in {0,1}* K(x) x + c The amount of information in x isn t much more than x Proof: For any string x, define a TM M x = On input w, overwrite w with x, and halt. Then M x on ε halts with just x on its tape. We have K(x) M x x + c 60

CS154, Lecture 12: Kolmogorov Complexity: A Universal Theory of Data Compression

CS154, Lecture 12: Kolmogorov Complexity: A Universal Theory of Data Compression CS154, Lecture 12: Kolmogorov Complexity: A Universal Theory of Data Compression Rosencrantz & Guildenstern Are Dead (Tom Stoppard) Rigged Lottery? And the winning numbers are: 1, 2, 3, 4, 5, 6 But is

More information

FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY

FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY 15-453 FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY KOLMOGOROV-CHAITIN (descriptive) COMPLEXITY TUESDAY, MAR 18 CAN WE QUANTIFY HOW MUCH INFORMATION IS IN A STRING? A = 01010101010101010101010101010101

More information

Finish K-Complexity, Start Time Complexity

Finish K-Complexity, Start Time Complexity 6.045 Finish K-Complexity, Start Time Complexity 1 Kolmogorov Complexity Definition: The shortest description of x, denoted as d(x), is the lexicographically shortest string such that M(w) halts

More information

Computability Theory

Computability Theory Computability Theory Cristian S. Calude May 2012 Computability Theory 1 / 1 Bibliography M. Sipser. Introduction to the Theory of Computation, PWS 1997. (textbook) Computability Theory 2 / 1 Supplementary

More information

,

, Kolmogorov Complexity Carleton College, CS 254, Fall 2013, Prof. Joshua R. Davis based on Sipser, Introduction to the Theory of Computation 1. Introduction Kolmogorov complexity is a theory of lossless

More information

CS154, Lecture 10: Rice s Theorem, Oracle Machines

CS154, Lecture 10: Rice s Theorem, Oracle Machines CS154, Lecture 10: Rice s Theorem, Oracle Machines Moral: Analyzing Programs is Really, Really Hard But can we more easily tell when some program analysis problem is undecidable? Problem 1 Undecidable

More information

Decidability and Undecidability

Decidability and Undecidability Decidability and Undecidability Major Ideas from Last Time Every TM can be converted into a string representation of itself. The encoding of M is denoted M. The universal Turing machine U TM accepts an

More information

CpSc 421 Homework 9 Solution

CpSc 421 Homework 9 Solution CpSc 421 Homework 9 Solution Attempt any three of the six problems below. The homework is graded on a scale of 100 points, even though you can attempt fewer or more points than that. Your recorded grade

More information

6.045: Automata, Computability, and Complexity Or, Great Ideas in Theoretical Computer Science Spring, Class 10 Nancy Lynch

6.045: Automata, Computability, and Complexity Or, Great Ideas in Theoretical Computer Science Spring, Class 10 Nancy Lynch 6.045: Automata, Computability, and Complexity Or, Great Ideas in Theoretical Computer Science Spring, 2010 Class 10 Nancy Lynch Today Final topic in computability theory: Self-Reference and the Recursion

More information

CISC 876: Kolmogorov Complexity

CISC 876: Kolmogorov Complexity March 27, 2007 Outline 1 Introduction 2 Definition Incompressibility and Randomness 3 Prefix Complexity Resource-Bounded K-Complexity 4 Incompressibility Method Gödel s Incompleteness Theorem 5 Outline

More information

Introduction to Turing Machines. Reading: Chapters 8 & 9

Introduction to Turing Machines. Reading: Chapters 8 & 9 Introduction to Turing Machines Reading: Chapters 8 & 9 1 Turing Machines (TM) Generalize the class of CFLs: Recursively Enumerable Languages Recursive Languages Context-Free Languages Regular Languages

More information

Decision Problems with TM s. Lecture 31: Halting Problem. Universe of discourse. Semi-decidable. Look at following sets: CSCI 81 Spring, 2012

Decision Problems with TM s. Lecture 31: Halting Problem. Universe of discourse. Semi-decidable. Look at following sets: CSCI 81 Spring, 2012 Decision Problems with TM s Look at following sets: Lecture 31: Halting Problem CSCI 81 Spring, 2012 Kim Bruce A TM = { M,w M is a TM and w L(M)} H TM = { M,w M is a TM which halts on input w} TOTAL TM

More information

Theory of Computation

Theory of Computation Theory of Computation Dr. Sarmad Abbasi Dr. Sarmad Abbasi () Theory of Computation 1 / 33 Lecture 20: Overview Incompressible strings Minimal Length Descriptions Descriptive Complexity Dr. Sarmad Abbasi

More information

Theory of Computation Lecture Notes. Problems and Algorithms. Class Information

Theory of Computation Lecture Notes. Problems and Algorithms. Class Information Theory of Computation Lecture Notes Prof. Yuh-Dauh Lyuu Dept. Computer Science & Information Engineering and Department of Finance National Taiwan University Problems and Algorithms c 2004 Prof. Yuh-Dauh

More information

CSE 105 Theory of Computation

CSE 105 Theory of Computation CSE 105 Theory of Computation http://www.jflap.org/jflaptmp/ Professor Jeanne Ferrante 1 Undecidability Today s Agenda Review and More Problems A Non-TR Language Reminders and announcements: HW 7 (Last!!)

More information

Lecture Notes: The Halting Problem; Reductions

Lecture Notes: The Halting Problem; Reductions Lecture Notes: The Halting Problem; Reductions COMS W3261 Columbia University 20 Mar 2012 1 Review Key point. Turing machines can be encoded as strings, and other Turing machines can read those strings

More information

Lecture 16: Time Complexity and P vs NP

Lecture 16: Time Complexity and P vs NP 6.045 Lecture 16: Time Complexity and P vs NP 1 Time-Bounded Complexity Classes Definition: TIME(t(n)) = { L there is a Turing machine M with time complexity O(t(n)) so that L = L(M) } = { L L is a language

More information

CSE 105 THEORY OF COMPUTATION

CSE 105 THEORY OF COMPUTATION CSE 105 THEORY OF COMPUTATION "Winter" 2018 http://cseweb.ucsd.edu/classes/wi18/cse105-ab/ Today's learning goals Sipser Ch 4.2 Trace high-level descriptions of algorithms for computational problems. Use

More information

Computability Theory. CS215, Lecture 6,

Computability Theory. CS215, Lecture 6, Computability Theory CS215, Lecture 6, 2000 1 The Birth of Turing Machines At the end of the 19th century, Gottlob Frege conjectured that mathematics could be built from fundamental logic In 1900 David

More information

Kolmogorov complexity and its applications

Kolmogorov complexity and its applications Spring, 2009 Kolmogorov complexity and its applications Paul Vitanyi Computer Science University of Amsterdam http://www.cwi.nl/~paulv/course-kc We live in an information society. Information science is

More information

Chomsky Normal Form and TURING MACHINES. TUESDAY Feb 4

Chomsky Normal Form and TURING MACHINES. TUESDAY Feb 4 Chomsky Normal Form and TURING MACHINES TUESDAY Feb 4 CHOMSKY NORMAL FORM A context-free grammar is in Chomsky normal form if every rule is of the form: A BC A a S ε B and C aren t start variables a is

More information

Peano Arithmetic. CSC 438F/2404F Notes (S. Cook) Fall, Goals Now

Peano Arithmetic. CSC 438F/2404F Notes (S. Cook) Fall, Goals Now CSC 438F/2404F Notes (S. Cook) Fall, 2008 Peano Arithmetic Goals Now 1) We will introduce a standard set of axioms for the language L A. The theory generated by these axioms is denoted PA and called Peano

More information

Kolmogorov complexity and its applications

Kolmogorov complexity and its applications CS860, Winter, 2010 Kolmogorov complexity and its applications Ming Li School of Computer Science University of Waterloo http://www.cs.uwaterloo.ca/~mli/cs860.html We live in an information society. Information

More information

ON COMPUTAMBLE NUMBERS, WITH AN APPLICATION TO THE ENTSCHENIDUGSPROBLEM. Turing 1936

ON COMPUTAMBLE NUMBERS, WITH AN APPLICATION TO THE ENTSCHENIDUGSPROBLEM. Turing 1936 ON COMPUTAMBLE NUMBERS, WITH AN APPLICATION TO THE ENTSCHENIDUGSPROBLEM Turing 1936 Where are We? Ignoramus et ignorabimus Wir mussen wissen Wir werden wissen We do not know We shall not know We must know

More information

Turing Machines Part III

Turing Machines Part III Turing Machines Part III Announcements Problem Set 6 due now. Problem Set 7 out, due Monday, March 4. Play around with Turing machines, their powers, and their limits. Some problems require Wednesday's

More information

The Turing machine model of computation

The Turing machine model of computation The Turing machine model of computation For most of the remainder of the course we will study the Turing machine model of computation, named after Alan Turing (1912 1954) who proposed the model in 1936.

More information

Lecture 23: Rice Theorem and Turing machine behavior properties 21 April 2009

Lecture 23: Rice Theorem and Turing machine behavior properties 21 April 2009 CS 373: Theory of Computation Sariel Har-Peled and Madhusudan Parthasarathy Lecture 23: Rice Theorem and Turing machine behavior properties 21 April 2009 This lecture covers Rice s theorem, as well as

More information

7.1 The Origin of Computer Science

7.1 The Origin of Computer Science CS125 Lecture 7 Fall 2016 7.1 The Origin of Computer Science Alan Mathison Turing (1912 1954) turing.jpg 170!201 pixels On Computable Numbers, with an Application to the Entscheidungsproblem 1936 1936:

More information

16.1 Countability. CS125 Lecture 16 Fall 2014

16.1 Countability. CS125 Lecture 16 Fall 2014 CS125 Lecture 16 Fall 2014 16.1 Countability Proving the non-existence of algorithms for computational problems can be very difficult. Indeed, we do not know how to prove P NP. So a natural question is

More information

Computational Models Lecture 9, Spring 2009

Computational Models Lecture 9, Spring 2009 Slides modified by Benny Chor, based on original slides by Maurice Herlihy, Brown University. p. 1 Computational Models Lecture 9, Spring 2009 Reducibility among languages Mapping reductions More undecidable

More information

Turing machines and linear bounded automata

Turing machines and linear bounded automata and linear bounded automata Informatics 2A: Lecture 29 John Longley School of Informatics University of Edinburgh jrl@inf.ed.ac.uk 25 November, 2011 1 / 13 1 The Chomsky hierarchy: summary 2 3 4 2 / 13

More information

FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY

FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY 15-453 FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY Chomsky Normal Form and TURING MACHINES TUESDAY Feb 4 CHOMSKY NORMAL FORM A context-free grammar is in Chomsky normal form if every rule is of the form:

More information

Computer Sciences Department

Computer Sciences Department Computer Sciences Department 1 Reference Book: INTRODUCTION TO THE THEORY OF COMPUTATION, SECOND EDITION, by: MICHAEL SIPSER Computer Sciences Department 3 ADVANCED TOPICS IN C O M P U T A B I L I T Y

More information

CS 154, Lecture 4: Limitations on DFAs (I), Pumping Lemma, Minimizing DFAs

CS 154, Lecture 4: Limitations on DFAs (I), Pumping Lemma, Minimizing DFAs CS 154, Lecture 4: Limitations on FAs (I), Pumping Lemma, Minimizing FAs Regular or Not? Non-Regular Languages = { w w has equal number of occurrences of 01 and 10 } REGULAR! C = { w w has equal number

More information

Undecidable Problems and Reducibility

Undecidable Problems and Reducibility University of Georgia Fall 2014 Reducibility We show a problem decidable/undecidable by reducing it to another problem. One type of reduction: mapping reduction. Definition Let A, B be languages over Σ.

More information

COS597D: Information Theory in Computer Science October 19, Lecture 10

COS597D: Information Theory in Computer Science October 19, Lecture 10 COS597D: Information Theory in Computer Science October 9, 20 Lecture 0 Lecturer: Mark Braverman Scribe: Andrej Risteski Kolmogorov Complexity In the previous lectures, we became acquainted with the concept

More information

Institute for Applied Information Processing and Communications (IAIK) Secure & Correct Systems. Decidability

Institute for Applied Information Processing and Communications (IAIK) Secure & Correct Systems. Decidability Decidability and the Undecidability of Predicate Logic IAIK Graz University of Technology georg.hofferek@iaik.tugraz.at 1 Fork of ways Brainteaser: Labyrinth Guards One to salvation One to perdition Two

More information

Lecture 20: conp and Friends, Oracles in Complexity Theory

Lecture 20: conp and Friends, Oracles in Complexity Theory 6.045 Lecture 20: conp and Friends, Oracles in Complexity Theory 1 Definition: conp = { L L NP } What does a conp computation look like? In NP algorithms, we can use a guess instruction in pseudocode:

More information

DRAFT. Diagonalization. Chapter 4

DRAFT. Diagonalization. Chapter 4 Chapter 4 Diagonalization..the relativized P =?NP question has a positive answer for some oracles and a negative answer for other oracles. We feel that this is further evidence of the difficulty of the

More information

1 Acceptance, Rejection, and I/O for Turing Machines

1 Acceptance, Rejection, and I/O for Turing Machines 1 Acceptance, Rejection, and I/O for Turing Machines Definition 1.1 (Initial Configuration) If M = (K,Σ,δ,s,H) is a Turing machine and w (Σ {, }) then the initial configuration of M on input w is (s, w).

More information

An example of a decidable language that is not a CFL Implementation-level description of a TM State diagram of TM

An example of a decidable language that is not a CFL Implementation-level description of a TM State diagram of TM Turing Machines Review An example of a decidable language that is not a CFL Implementation-level description of a TM State diagram of TM Varieties of TMs Multi-Tape TMs Nondeterministic TMs String Enumerators

More information

CS 361 Meeting 26 11/10/17

CS 361 Meeting 26 11/10/17 CS 361 Meeting 26 11/10/17 1. Homework 8 due Announcements A Recognizable, but Undecidable Language 1. Last class, I presented a brief, somewhat inscrutable proof that the language A BT M = { M w M is

More information

CSCE 551: Chin-Tser Huang. University of South Carolina

CSCE 551: Chin-Tser Huang. University of South Carolina CSCE 551: Theory of Computation Chin-Tser Huang huangct@cse.sc.edu University of South Carolina Church-Turing Thesis The definition of the algorithm came in the 1936 papers of Alonzo Church h and Alan

More information

Models. Models of Computation, Turing Machines, and the Limits of Turing Computation. Effective Calculability. Motivation for Models of Computation

Models. Models of Computation, Turing Machines, and the Limits of Turing Computation. Effective Calculability. Motivation for Models of Computation Turing Computation /0/ Models of Computation, Turing Machines, and the Limits of Turing Computation Bruce MacLennan Models A model is a tool intended to address a class of questions about some domain of

More information

Lecture 12: Mapping Reductions

Lecture 12: Mapping Reductions Lecture 12: Mapping Reductions October 18, 2016 CS 1010 Theory of Computation Topics Covered 1. The Language EQ T M 2. Mapping Reducibility 3. The Post Correspondence Problem 1 The Language EQ T M The

More information

Lecture 22: PSPACE

Lecture 22: PSPACE 6.045 Lecture 22: PSPACE 1 VOTE VOTE VOTE For your favorite course on automata and complexity Please complete the online subject evaluation for 6.045 2 Final Exam Information Who: You On What: Everything

More information

Turing Machines (TM) The Turing machine is the ultimate model of computation.

Turing Machines (TM) The Turing machine is the ultimate model of computation. TURING MACHINES Turing Machines (TM) The Turing machine is the ultimate model of computation. Alan Turing (92 954), British mathematician/engineer and one of the most influential scientists of the last

More information

Lecture 24: Randomized Complexity, Course Summary

Lecture 24: Randomized Complexity, Course Summary 6.045 Lecture 24: Randomized Complexity, Course Summary 1 1/4 1/16 1/4 1/4 1/32 1/16 1/32 Probabilistic TMs 1/16 A probabilistic TM M is a nondeterministic TM where: Each nondeterministic step is called

More information

Final Exam Comments. UVa - cs302: Theory of Computation Spring < Total

Final Exam Comments. UVa - cs302: Theory of Computation Spring < Total UVa - cs302: Theory of Computation Spring 2008 Final Exam Comments < 50 50 59 60 69 70 79 80 89 90 94 95-102 Total 2 6 8 22 16 16 12 Problem 1: Short Answers. (20) For each question, provide a correct,

More information

Further discussion of Turing machines

Further discussion of Turing machines Further discussion of Turing machines In this lecture we will discuss various aspects of decidable and Turing-recognizable languages that were not mentioned in previous lectures. In particular, we will

More information

Recovery Based on Kolmogorov Complexity in Underdetermined Systems of Linear Equations

Recovery Based on Kolmogorov Complexity in Underdetermined Systems of Linear Equations Recovery Based on Kolmogorov Complexity in Underdetermined Systems of Linear Equations David Donoho Department of Statistics Stanford University Email: donoho@stanfordedu Hossein Kakavand, James Mammen

More information

CS154. Non-Regular Languages, Minimizing DFAs

CS154. Non-Regular Languages, Minimizing DFAs CS54 Non-Regular Languages, Minimizing FAs CS54 Homework is due! Homework 2 will appear this afternoon 2 The Pumping Lemma: Structure in Regular Languages Let L be a regular language Then there is a positive

More information

(a) Definition of TMs. First Problem of URMs

(a) Definition of TMs. First Problem of URMs Sec. 4: Turing Machines First Problem of URMs (a) Definition of the Turing Machine. (b) URM computable functions are Turing computable. (c) Undecidability of the Turing Halting Problem That incrementing

More information

Definition: conp = { L L NP } What does a conp computation look like?

Definition: conp = { L L NP } What does a conp computation look like? Space Complexity 28 Definition: conp = { L L NP } What does a conp computation look like? In NP algorithms, we can use a guess instruction in pseudocode: Guess string y of x k length and the machine accepts

More information

Introduction to Kolmogorov Complexity

Introduction to Kolmogorov Complexity Introduction to Kolmogorov Complexity Marcus Hutter Canberra, ACT, 0200, Australia http://www.hutter1.net/ ANU Marcus Hutter - 2 - Introduction to Kolmogorov Complexity Abstract In this talk I will give

More information

Computational Complexity: A Modern Approach. Draft of a book: Dated January 2007 Comments welcome!

Computational Complexity: A Modern Approach. Draft of a book: Dated January 2007 Comments welcome! i Computational Complexity: A Modern Approach Draft of a book: Dated January 2007 Comments welcome! Sanjeev Arora and Boaz Barak Princeton University complexitybook@gmail.com Not to be reproduced or distributed

More information

Turing s thesis: (1930) Any computation carried out by mechanical means can be performed by a Turing Machine

Turing s thesis: (1930) Any computation carried out by mechanical means can be performed by a Turing Machine Turing s thesis: (1930) Any computation carried out by mechanical means can be performed by a Turing Machine There is no known model of computation more powerful than Turing Machines Definition of Algorithm:

More information

A version of for which ZFC can not predict a single bit Robert M. Solovay May 16, Introduction In [2], Chaitin introd

A version of for which ZFC can not predict a single bit Robert M. Solovay May 16, Introduction In [2], Chaitin introd CDMTCS Research Report Series A Version of for which ZFC can not Predict a Single Bit Robert M. Solovay University of California at Berkeley CDMTCS-104 May 1999 Centre for Discrete Mathematics and Theoretical

More information

Notes on Complexity Theory Last updated: November, Lecture 10

Notes on Complexity Theory Last updated: November, Lecture 10 Notes on Complexity Theory Last updated: November, 2015 Lecture 10 Notes by Jonathan Katz, lightly edited by Dov Gordon. 1 Randomized Time Complexity 1.1 How Large is BPP? We know that P ZPP = RP corp

More information

Post s Correspondence Problem (PCP) Is Undecidable. Presented By Bharath Bhushan Reddy Goulla

Post s Correspondence Problem (PCP) Is Undecidable. Presented By Bharath Bhushan Reddy Goulla Post s Correspondence Problem (PCP) Is Undecidable Presented By Bharath Bhushan Reddy Goulla Contents : Introduction What Is PCP? Instances Of PCP? Introduction Of Modified PCP (MPCP) Why MPCP? Reducing

More information

Theory of Computation

Theory of Computation Theory of Computation Dr. Sarmad Abbasi Dr. Sarmad Abbasi () Theory of Computation / Lecture 3: Overview Decidability of Logical Theories Presburger arithmetic Decidability of Presburger Arithmetic Dr.

More information

Creative Objectivism, a powerful alternative to Constructivism

Creative Objectivism, a powerful alternative to Constructivism Creative Objectivism, a powerful alternative to Constructivism Copyright c 2002 Paul P. Budnik Jr. Mountain Math Software All rights reserved Abstract It is problematic to allow reasoning about infinite

More information

Nondeterministic Finite Automata

Nondeterministic Finite Automata Nondeterministic Finite Automata Mahesh Viswanathan Introducing Nondeterminism Consider the machine shown in Figure. Like a DFA it has finitely many states and transitions labeled by symbols from an input

More information

Turing machines and computable functions

Turing machines and computable functions Turing machines and computable functions In this chapter we address the question: how can we define the class of discrete functions that are computable? Probably, most of you are familiar with a few programming

More information

6.045 Final Exam Solutions

6.045 Final Exam Solutions 6.045J/18.400J: Automata, Computability and Complexity Prof. Nancy Lynch, Nati Srebro 6.045 Final Exam Solutions May 18, 2004 Susan Hohenberger Name: Please write your name on each page. This exam is open

More information

Computability and Complexity Theory

Computability and Complexity Theory Discrete Math for Bioinformatics WS 09/10:, by A Bockmayr/K Reinert, January 27, 2010, 18:39 9001 Computability and Complexity Theory Computability and complexity Computability theory What problems can

More information

cse303 ELEMENTS OF THE THEORY OF COMPUTATION Professor Anita Wasilewska

cse303 ELEMENTS OF THE THEORY OF COMPUTATION Professor Anita Wasilewska cse303 ELEMENTS OF THE THEORY OF COMPUTATION Professor Anita Wasilewska LECTURE 13 CHAPTER 4 TURING MACHINES 1. The definition of Turing machine 2. Computing with Turing machines 3. Extensions of Turing

More information

Nondeterministic finite automata

Nondeterministic finite automata Lecture 3 Nondeterministic finite automata This lecture is focused on the nondeterministic finite automata (NFA) model and its relationship to the DFA model. Nondeterminism is an important concept in the

More information

Computability Theory

Computability Theory Computability Theory Cristian S. Calude and Nicholas J. Hay May June 2009 Computability Theory 1 / 155 1 Register machines 2 Church-Turing thesis 3 Decidability 4 Reducibility 5 A definition of information

More information

Turing Machines Part II

Turing Machines Part II Turing Machines Part II Hello Hello Condensed Slide Slide Readers! Readers! This This lecture lecture is is almost almost entirely entirely animations that that show show how how each each Turing Turing

More information

Theory of Computation

Theory of Computation Theory of Computation Lecture #10 Sarmad Abbasi Virtual University Sarmad Abbasi (Virtual University) Theory of Computation 1 / 43 Lecture 10: Overview Linear Bounded Automata Acceptance Problem for LBAs

More information

Primitive recursive functions: decidability problems

Primitive recursive functions: decidability problems Primitive recursive functions: decidability problems Armando B. Matos October 24, 2014 Abstract Although every primitive recursive (PR) function is total, many problems related to PR functions are undecidable.

More information

Busch Complexity Lectures: Turing s Thesis. Costas Busch - LSU 1

Busch Complexity Lectures: Turing s Thesis. Costas Busch - LSU 1 Busch Complexity Lectures: Turing s Thesis Costas Busch - LSU 1 Turing s thesis (1930): Any computation carried out by mechanical means can be performed by a Turing Machine Costas Busch - LSU 2 Algorithm:

More information

Notes on Complexity Theory Last updated: October, Lecture 6

Notes on Complexity Theory Last updated: October, Lecture 6 Notes on Complexity Theory Last updated: October, 2015 Lecture 6 Notes by Jonathan Katz, lightly edited by Dov Gordon 1 PSPACE and PSPACE-Completeness As in our previous study of N P, it is useful to identify

More information

FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY

FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY 15-453 FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY REVIEW for MIDTERM 1 THURSDAY Feb 6 Midterm 1 will cover everything we have seen so far The PROBLEMS will be from Sipser, Chapters 1, 2, 3 It will be

More information

Homework 8. a b b a b a b. two-way, read/write

Homework 8. a b b a b a b. two-way, read/write Homework 8 309 Homework 8 1. Describe a TM that accepts the set {a n n is a power of 2}. Your description should be at the level of the descriptions in Lecture 29 of the TM that accepts {ww w Σ } and the

More information

Theory of Computation

Theory of Computation Theory of Computation (Feodor F. Dragan) Department of Computer Science Kent State University Spring, 2018 Theory of Computation, Feodor F. Dragan, Kent State University 1 Before we go into details, what

More information

1 Deterministic Turing Machines

1 Deterministic Turing Machines Time and Space Classes Exposition by William Gasarch 1 Deterministic Turing Machines Turing machines are a model of computation. It is believed that anything that can be computed can be computed by a Turing

More information

Stochasticity in Algorithmic Statistics for Polynomial Time

Stochasticity in Algorithmic Statistics for Polynomial Time Stochasticity in Algorithmic Statistics for Polynomial Time Alexey Milovanov 1 and Nikolay Vereshchagin 2 1 National Research University Higher School of Economics, Moscow, Russia almas239@gmail.com 2

More information

Notes on State Minimization

Notes on State Minimization U.C. Berkeley CS172: Automata, Computability and Complexity Handout 1 Professor Luca Trevisan 2/3/2015 Notes on State Minimization These notes present a technique to prove a lower bound on the number of

More information

Problems, and How Computer Scientists Solve Them Manas Thakur

Problems, and How Computer Scientists Solve Them Manas Thakur Problems, and How Computer Scientists Solve Them PACE Lab, IIT Madras Content Credits Introduction to Automata Theory, Languages, and Computation, 3rd edition. Hopcroft et al. Introduction to the Theory

More information

Definition: Alternating time and space Game Semantics: State of machine determines who

Definition: Alternating time and space Game Semantics: State of machine determines who CMPSCI 601: Recall From Last Time Lecture Definition: Alternating time and space Game Semantics: State of machine determines who controls, White wants it to accept, Black wants it to reject. White wins

More information

Chapter 2 Algorithms and Computation

Chapter 2 Algorithms and Computation Chapter 2 Algorithms and Computation In this chapter, we first discuss the principles of algorithm and computation in general framework, common both in classical and quantum computers, then we go to the

More information

Lecture 23: More PSPACE-Complete, Randomized Complexity

Lecture 23: More PSPACE-Complete, Randomized Complexity 6.045 Lecture 23: More PSPACE-Complete, Randomized Complexity 1 Final Exam Information Who: You On What: Everything through PSPACE (today) With What: One sheet (double-sided) of notes are allowed When:

More information

Theory of computation: initial remarks (Chapter 11)

Theory of computation: initial remarks (Chapter 11) Theory of computation: initial remarks (Chapter 11) For many purposes, computation is elegantly modeled with simple mathematical objects: Turing machines, finite automata, pushdown automata, and such.

More information

Turing Machines Part II

Turing Machines Part II Turing Machines Part II Problem Set Set Five Five due due in in the the box box up up front front using using a late late day. day. Hello Hello Condensed Slide Slide Readers! Readers! This This lecture

More information

Lecture 12: Interactive Proofs

Lecture 12: Interactive Proofs princeton university cos 522: computational complexity Lecture 12: Interactive Proofs Lecturer: Sanjeev Arora Scribe:Carl Kingsford Recall the certificate definition of NP. We can think of this characterization

More information

Automata and Computability. Solutions to Exercises

Automata and Computability. Solutions to Exercises Automata and Computability Solutions to Exercises Spring 27 Alexis Maciel Department of Computer Science Clarkson University Copyright c 27 Alexis Maciel ii Contents Preface vii Introduction 2 Finite Automata

More information

Lecture 2. 1 More N P-Compete Languages. Notes on Complexity Theory: Fall 2005 Last updated: September, Jonathan Katz

Lecture 2. 1 More N P-Compete Languages. Notes on Complexity Theory: Fall 2005 Last updated: September, Jonathan Katz Notes on Complexity Theory: Fall 2005 Last updated: September, 2005 Jonathan Katz Lecture 2 1 More N P-Compete Languages It will be nice to find more natural N P-complete languages. To that end, we ine

More information

The axiomatic power of Kolmogorov complexity

The axiomatic power of Kolmogorov complexity The axiomatic power of Kolmogorov complexity Laurent Bienvenu 1, Andrei Romashchenko 2, Alexander Shen 2, Antoine Taveneaux 1, and Stijn Vermeeren 3 1 LIAFA, CNRS & Université Paris 7 2 LIRMM, CNRS & Université

More information

6.045: Automata, Computability, and Complexity (GITCS) Class 17 Nancy Lynch

6.045: Automata, Computability, and Complexity (GITCS) Class 17 Nancy Lynch 6.045: Automata, Computability, and Complexity (GITCS) Class 17 Nancy Lynch Today Probabilistic Turing Machines and Probabilistic Time Complexity Classes Now add a new capability to standard TMs: random

More information

Turing machines Finite automaton no storage Pushdown automaton storage is a stack What if we give the automaton a more flexible storage?

Turing machines Finite automaton no storage Pushdown automaton storage is a stack What if we give the automaton a more flexible storage? Turing machines Finite automaton no storage Pushdown automaton storage is a stack What if we give the automaton a more flexible storage? What is the most powerful of automata? In this lecture we will introduce

More information

CSCI3390-Assignment 2 Solutions

CSCI3390-Assignment 2 Solutions CSCI3390-Assignment 2 Solutions due February 3, 2016 1 TMs for Deciding Languages Write the specification of a Turing machine recognizing one of the following three languages. Do one of these problems.

More information

CS 275 Automata and Formal Language Theory

CS 275 Automata and Formal Language Theory CS 275 Automata and Formal Language Theory Course Notes Part III: Limits of Computation Chapter III.1: Introduction Anton Setzer http://www.cs.swan.ac.uk/ csetzer/lectures/ automataformallanguage/current/index.html

More information

Poincaré, Heisenberg, Gödel. Some limits of scientific knowledge. Fernando Sols Universidad Complutense de Madrid

Poincaré, Heisenberg, Gödel. Some limits of scientific knowledge. Fernando Sols Universidad Complutense de Madrid Poincaré, Heisenberg, Gödel. Some limits of scientific knowledge. Fernando Sols Universidad Complutense de Madrid Henry Poincaré (1854-1912) nonlinear dynamics Werner Heisenberg (1901-1976) uncertainty

More information

Turing Machines Part Two

Turing Machines Part Two Turing Machines Part Two Recap from Last Time Our First Turing Machine q acc a start q 0 q 1 a This This is is the the Turing Turing machine s machine s finiteisttiteiconntont. finiteisttiteiconntont.

More information

The Legacy of Hilbert, Gödel, Gentzen and Turing

The Legacy of Hilbert, Gödel, Gentzen and Turing The Legacy of Hilbert, Gödel, Gentzen and Turing Amílcar Sernadas Departamento de Matemática - Instituto Superior Técnico Security and Quantum Information Group - Instituto de Telecomunicações TULisbon

More information

Kolmogorov Complexity in Randomness Extraction

Kolmogorov Complexity in Randomness Extraction LIPIcs Leibniz International Proceedings in Informatics Kolmogorov Complexity in Randomness Extraction John M. Hitchcock, A. Pavan 2, N. V. Vinodchandran 3 Department of Computer Science, University of

More information

On Rice s theorem. Hans Hüttel. October 2001

On Rice s theorem. Hans Hüttel. October 2001 On Rice s theorem Hans Hüttel October 2001 We have seen that there exist languages that are Turing-acceptable but not Turing-decidable. An important example of such a language was the language of the Halting

More information