Lecture 13: Foundations of Math and Kolmogorov Complexity

Size: px
Start display at page:

Download "Lecture 13: Foundations of Math and Kolmogorov Complexity"

Transcription

1 6.045 Lecture 13: Foundations of Math and Kolmogorov Complexity 1

2 Self-Reference and the Recursion Theorem 2

3 Lemma: There is a computable function q : Σ* Σ* such that for every string w, q(w) is the description of a TM P w that on every input, prints out w and then accepts Proof Define a TM Q: s w Q P w w 3

4 Theorem: There is a Self-Printing TM Proof: First define a TM B which does this: M B w P M M M Now consider the TM that looks like this: w B w P B B P B B B No explicit self-reference here! QED 4

5 The Recursion Theorem Theorem: For every TM T computing a function t : Σ* Σ* Σ* there is a Turing machine R computing a function r : Σ* Σ*, such that for every string w, r(w) = t(r, w) (a,b) T t(a,b) w R t(r,w) 5

6 Proof: (a,b) T t(a,b) Define M = N B N w B Y w T w P N N w N Define R: What is S? w P M M B S w T t(s,w) 6

7 Proof: (a,b) Define M = T M t(a,b) B What is M(M,w)? N w B Y w T w P M M w M Define R: S M S w P B T t(s,w) M w 7

8 Proof: (a,b) T t(a,b) M B w M PM w B Y w T Define R: S S = Y = R. QED M S w P B T t(s,w) M w 8

9 FOO x (y) := Output x and halt. BAR(M) := Output N(w) = Run FOO M outputting M. Run M on (M, w) Q(N, w) := Run BAR(N) outputting S. Run T on (S, w) R(w) := Run FOO Q outputting Q. Run BAR(Q) outputting S. Run T on (S, x) Claim: S is a description of R itself! S(w) = Run FOO Q outputting Q. Run Q on (Q, w)

10 FOO x (y) := Output x and halt. BAR(M) := Output N(w) = Run FOO M outputting M. Run M on (M, w) Q(N, w) := Run BAR(N) outputting S. Run T on (S, w) R(w) := Run FOO Q outputting Q. Run BAR(Q) outputting S. Run T on (S, w) Claim: S is a description of R itself! S(w) = Run FOO Q outputting Q. Run BAR(Q) outputting S. Run T on (S, w) Therefore R(w) = T(R, w)

11 For every computable t, there is a computable r such that r(w) = t(r,w) where R is a description of r Suppose we can design a TM T of the form: On input (x,w), do bla bla with x, do bla bla bla with w, etc. etc. We can then find a TM R with the behavior: On input w, do bla bla with R, do bla bla bla with w, etc. etc. We can use the operation: Obtain your own description in Turing machine pseudocode! 11

12 Theorem: A TM is undecidable Proof (using the recursion theorem) Assume H decides A TM Construct machine B such that on input w: 1. Obtains its own description B 2. Runs H on (B, w) and flips the output Running B on input w always does the opposite of what H says it should! A formalization of free will paradoxes! No single machine can predict behavior of all others 12

13 Turing Machine Minimization MIN = {M M is a minimal-state Turing machine} Theorem: MIN is unrecognizable! Proof: Suppose we could recognize MIN with TM M M(x) := Obtain description of M. For k = 1,2,, Run M on the TMs M 1, M k for k steps, until M accepts some M i with Q(M i ) > Q(M). Output M i (x). We have: 1. L M = L M i [by construction] 2. M has fewer states than M i 3. M i is minimal [by definition of MIN] CONTRADICTION! 13

14 Computability and the Foundations of Mathematics 17

15 Formal Systems of Mathematics A formal system describes a formal language for - writing (finite) mathematical statements, - has a definition of a proof of a statement Example: Every TM M defines some formal system F - {Mathematical statements in F } = * String w represents the statement M halts on w - A proof that M halts on w can be defined to be the computation history of M on w: the sequence of configurations C 0 C 1 C t that M goes through while computing on w Could sometimes prove M doesn t halt on w 18

16 Interesting Systems of Mathematics Define a formal system F to be interesting if: 1. Any mathematical statement about computation can be (computably) described as a statement of F. Given (M, w), there is a (computable) S M,w of F such that S M,w is true in F if and only if M accepts w. 2. Proofs are convincing a TM can check that a candidate proof of a theorem is correct This set is decidable: {(S, P) P is a proof of S in F } 3. If S is in F and there is a proof of S describable as a computation, then there s a proof of S in F. If M halts on w, then there s either a proof P of S M,w or a proof P of S M,w 19

17 Consistency and Completeness A formal system F is inconsistent if there is a statement S in F such that both S and S are provable in F F is consistent if it is NOT inconsistent A formal system F is incomplete if there is a statement S in F such that neither S nor S are provable in F F is complete if it is NOT incomplete 20

18 Limitations on Mathematics! For every consistent and interesting F, Theorem 1. (Gödel 1931) F must be incomplete! There are mathematical statements that are true but cannot be proved. Theorem 2. (Gödel 1931) The consistency of F cannot be proved in F. Theorem 3. (Church-Turing 1936) The problem of checking whether a given statement in F has a proof is undecidable. 21

19 Unprovable Truths in Mathematics (Gödel) Every consistent interesting F is incomplete: there are statements that cannot be proved or disproved. Let S M, w in F be true if and only if M accepts w Proof: Define TM G(w): 1. Obtain own description G [Recursion Theorem!] 2. For all strings P in lexicographical order, If (P is a proof of S G, w in F ) then reject If (P is a proof of S G, w in F ) then accept Note: If F is complete then G cannot run forever! 1. If (G accepts w) then have proof P of G doesn t accept w 2. If (G rejects w) then found proof P of G accepts w In either case, F is inconsistent! Proof of S G, w and S G, w 22

20 (Gödel 1931) The consistency of F cannot be proved within any interesting consistent F Proof: Assume we can prove F is consistent in F We constructed :S G, w = G does not accept w which is true, but has no proof in F G does not accept w There is no proof of :S G, w in F But if there s a proof of F is consistent in F, then there is a proof of :S G, w in F (here s the proof): If S G, w is true, then there is a proof in F of S G, w and a proof in in F of :S G, w But since F is consistent, this cannot be true. Therefore, :S G, w is true This contradicts the previous theorem! 23

21 Undecidability in Mathematics PROVABLE F = {S there s a proof in F of S, or there s a proof in F of :S} (Church-Turing 1936) For every interesting consistent F, PROVABLE F is undecidable Proof: Suppose PROVABLE F is decidable with TM P. Then we can decide A TM with the following procedure: On input (M, w), run the TM P on input S M,w If P accepts, examine all proofs in lex order If a proof of S M,w is found then accept If a proof of :S M,w is found then reject If P rejects, then reject. Why does this work? 24

22 Kolmogorov Complexity: A Universal Theory of Data Compression 25

23 The Church-Turing Thesis: Everyone s Intuitive Notion of Algorithms = Turing Machines This is not a theorem it is a falsifiable scientific hypothesis. A Universal Theory of Computation 26

24 A Universal Theory of Information? Can we quantify how much information is contained in a string? A = B = Idea: The more we can compress a string, the less information it contains. 27

25 Information as Description Thesis: The amount of information in a string x is the length of the shortest description of x Let x 2 {0,1}* How should we describe strings? Use Turing machines with inputs! Def: A description of x is a string <M,w> such that M on input w halts with only x on its tape. Def: The shortest description of x, denoted as d(x), is the lexicographically shortest string <M,w> such that M(w) halts with only x on its tape. 28

26 A Specific Pairing Function Theorem. There is a 1-1 computable function <,> : Σ* x Σ* Σ* and computable functions 1 and 2 : Σ* Σ* such that: z = <M,w> iff 1 (z) = M and 2 (z) = w Define: <M,w> := 0 M 1 M w (Example: <10110,101> = ) Note that <M,w> = 2 M + w

27 Kolmogorov Complexity (1960 s) Definition: The shortest description of x, denoted as d(x), is the lexicographically shortest string <M,w> such that M(w) halts with only x on its tape. Definition: The Kolmogorov complexity of x, denoted as K(x), is d(x). EXAMPLES?? Let s first determine some properties of K. Examples will fall out of this. 30

28 A Simple Upper Bound Theorem: There is a fixed c so that for all x in {0,1}* K(x) x + c The amount of information in x isn t much more than x Proof: Define a TM M = On input w, halt. On any string x, M(x) halts with x on its tape. Observe that <M,x> is a description of x. Let c = 2 M +1 Then K(x) <M,x> 2 M + x + 1 x + c 31

29 Repetitive Strings have Low K-Complexity Theorem: There is a fixed c so that for all n 2, and all x ϵ {0,1}*, K(x n ) K(x) + c log n The information in x n isn t much more than that in x Proof: Define the TM N = On input <n,<m,w>>, Let x = M(w). Print x for n times. Let <M,w> be the shortest description of x. Then K(x n ) K(<N,<n,<M,w>>>) 2 N + d log n + K(x) c log n + K(x) for some constants c and d 32

30 Repetitive Strings have Low K-Complexity Theorem: There is a fixed c so that for all n 2, and all x ϵ {0,1}*, K(x n ) K(x) + c log n The information in x n isn t much more than that in x Recall: A = For w = (01) n, we have K(w) K(01) + c log n So for all n, K((01) n ) d + c log n for a fixed c, d 33

31 Does The Computational Model Matter? Turing machines are one programming language. If we use other programming languages, could we get significantly shorter descriptions? An interpreter is a semi-computable function p : Σ* Σ* Takes programs as input, and (may) print their outputs Definition: Let x ϵ {0,1}*. The shortest description of x under p, called d p (x), is the lexicographically shortest string w for which p(w) = x. Definition: The K p complexity of x is K p (x) := d p (x). 34

32 Does The Computational Model Matter? Theorem: For every interpreter p, there is a fixed c so that for all x ϵ {0,1}*, K(x) K p (x) + c Moral: Using another programming language would only change K(x) by some additive constant Proof: Define M = On w, simulate p(w) and write its output to tape Then <M,d p (x)> is a description of x, so K(x) <M,d p (x)> 2 M + K p (x) + 1 c + K p (x) 35

33 There Exist Incompressible Strings Theorem: For all n, there is an x ϵ {0,1} n such that K(x) n There are incompressible strings of every length Proof: (Number of binary strings of length n) = 2 n but (Number of descriptions of length < n) (Number of binary strings of length < n) = n-1 = 2 n 1 Therefore, there is at least one n-bit string x that does not have a description of length < n 36

34 Random Strings Are Incompressible! Theorem: For all n and c 1, Pr x ϵ {0,1} n[ K(x) n-c ] 1 1/2 c Most strings are highly incompressible Proof: (Number of binary strings of length n) = 2 n but (Number of descriptions of length < n-c) (Number of binary strings of length < n-c) = 2 n-c 1 Hence the probability that a random x satisfies K(x) < n-c is at most (2 n-c 1)/2 n < 1/2 c. 37

35 KOLMOGOROV DIRECTIONS 38

36 Kolmogorov Complexity: Try it! Give short algorithms for generating the strings:

37 Kolmogorov Complexity: Try it! Give short algorithms for generating the strings:

38 Kolmogorov Complexity: Try it! Give short algorithms for generating the strings:

39 Kolmogorov Complexity: Try it! Give short algorithms for generating the strings: This seems hard to determine in general. Why? 42

40 Computing Compressibility? Can an algorithm perform optimal compression? Can algorithms tell us if a given string is compressible? COMPRESS = { (x,c) K(x) c} Theorem: COMPRESS is undecidable! Idea: If decidable, we could design an algorithm that prints the shortest incompressible string of length n But such a string could then be succinctly described, by providing the algorithm code and n in binary! Berry Paradox: The smallest integer that cannot be defined in less than thirteen words. 43

41 Computing Compressibility? COMPRESS = {(x,c) K(x) c} Theorem: COMPRESS is undecidable! Proof: Suppose it s decidable. Consider the TM: M = On input x ϵ {0,1}*, let N = 2 x. For all y ϵ {0,1}* in lexicographical order, If (y,n) COMPRESS then print y and halt. M(x) prints the shortest string y with K(y ) > 2 x. <M,x> is a description of y, and <M,x> d + x So 2 x < K(y ) d + x. CONTRADICTION for large x! 44

42 Yet Another Proof that A TM is Undecidable! COMPRESS = {(x,c) K(x) c} Theorem: A TM is undecidable. Proof: Reduction from COMPRESS to A TM. Given a pair (x,c), our reduction constructs a TM: M x,c = On input w, For all pairs <M,w > with <M,w > c, simulate each M on w in parallel. If some M halts and prints x, then accept. K(x) c M x,c accepts ε 45

43 More on Interesting Formal Systems A formal system F is interesting if it is finite and: 1. Any mathematical statement about computation can also be effectively described within F. For all strings x and integers c, there is a S x,c in F that is equivalent to K(x) c 2. Proofs are convincing: it should be possible to check that a proof of a theorem is correct This set is decidable: { (S,P) P is a proof of S in F } 46

44 The Unprovable Truth About K-Complexity Theorem: For every interesting consistent F, There is a t s.t. for all x, K(x) > t is unprovable in F Proof: Define an M that treats its input as an integer: M(k) := Search over all strings x and proofs P for a proof P in F that K(x) > k. Output x if found Suppose M(k) halts. It must print some output x Then K(x ) = K(<M,k>) c + k c + log k for some c Because F is consistent, K(x ) > k is true But k < c + log k only holds for small enough k If we choose t to be greater than these k then M(t) cannot halt, so K(x) > t has no proof! 47

45 Random Unprovable Truths Theorem: For every interesting consistent F, There is a t s.t. for all x, K(x) > t is unprovable in F For a randomly chosen x of length t+100, K(x) > t is true with probability at least 1-1/2 100 We can randomly generate true statements in F which have no proof in F, with high probability! For every interesting formal system F there is always some finite integer (say, t=10000) so that you ll never be able to prove in F that a random bit string requires a bit program! 48

46 Proving Theorems With K-Complexity Theorem: L = {x x x 0, 1 } is not regular. Proof: Suppose L is recognized by a DFA D. Let n 0 and choose an x 0, 1 such that K x n Let q x be the state of D reached after reading in x Define a TM M(D, q, n): Find a path P in D of length n that starts from state q and ends in an accept state. Print the n-bit string along path P, and halt. Claim: The pair <M,(D, q x, n)> is a description of x! So n K x <M,(D, q x, n)> O log n CONTRADICTION! 50

47 Next Episode: Complexity Theory! 51

48 Repetitive Strings have Low Information Theorem: There is a fixed c so that for all x ϵ {0,1}* K(xx) K(x) + c The information in xx isn t much more than that in x Proof: Let N = On <M,w>, let s = M(w). Print ss. Suppose <M,w> is the shortest description of x. Then <N,<M,w>> is a description of xx Therefore K(xx) <N,<M,w>> 2 N + <M,w> N + K(x) + 1 c + K(x) 57

49 Information as Description Thesis: The amount of information in a string x is the length of the smallest description of x Let x 2 {0,1}* How should we describe strings? Use Turing Machines! Def. The shortest description of x, denoted d(x), is the lexicographically shortest string M such that the TM M on input ε halts with only x on its tape. 58

50 Kolmogorov Complexity (1960 s) Def. The shortest description of x, denoted d(x), is the lexicographically shortest string M such that the TM M on input ε halts with only x on its tape. Definition: The Kolmogorov complexity of x, denoted as K(x), is d(x). EXAMPLES?? Let s first determine some properties of K. Examples will fall out of this. 59

51 A Simple Upper Bound Theorem: There is a fixed c so that for all x in {0,1}* K(x) x + c The amount of information in x isn t much more than x Proof: For any string x, define a TM M x = On input w, overwrite w with x, and halt. Then M x on ε halts with just x on its tape. We have K(x) M x x + c 60

CS154, Lecture 12: Kolmogorov Complexity: A Universal Theory of Data Compression

CS154, Lecture 12: Kolmogorov Complexity: A Universal Theory of Data Compression CS154, Lecture 12: Kolmogorov Complexity: A Universal Theory of Data Compression Rosencrantz & Guildenstern Are Dead (Tom Stoppard) Rigged Lottery? And the winning numbers are: 1, 2, 3, 4, 5, 6 But is

More information

FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY

FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY 15-453 FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY KOLMOGOROV-CHAITIN (descriptive) COMPLEXITY TUESDAY, MAR 18 CAN WE QUANTIFY HOW MUCH INFORMATION IS IN A STRING? A = 01010101010101010101010101010101

More information

Finish K-Complexity, Start Time Complexity

Finish K-Complexity, Start Time Complexity 6.045 Finish K-Complexity, Start Time Complexity 1 Kolmogorov Complexity Definition: The shortest description of x, denoted as d(x), is the lexicographically shortest string such that M(w) halts

More information

Computability Theory

Computability Theory Computability Theory Cristian S. Calude May 2012 Computability Theory 1 / 1 Bibliography M. Sipser. Introduction to the Theory of Computation, PWS 1997. (textbook) Computability Theory 2 / 1 Supplementary

More information

,

, Kolmogorov Complexity Carleton College, CS 254, Fall 2013, Prof. Joshua R. Davis based on Sipser, Introduction to the Theory of Computation 1. Introduction Kolmogorov complexity is a theory of lossless

More information

CS154, Lecture 10: Rice s Theorem, Oracle Machines

CS154, Lecture 10: Rice s Theorem, Oracle Machines CS154, Lecture 10: Rice s Theorem, Oracle Machines Moral: Analyzing Programs is Really, Really Hard But can we more easily tell when some program analysis problem is undecidable? Problem 1 Undecidable

More information

CpSc 421 Homework 9 Solution

CpSc 421 Homework 9 Solution CpSc 421 Homework 9 Solution Attempt any three of the six problems below. The homework is graded on a scale of 100 points, even though you can attempt fewer or more points than that. Your recorded grade

More information

Introduction to Turing Machines. Reading: Chapters 8 & 9

Introduction to Turing Machines. Reading: Chapters 8 & 9 Introduction to Turing Machines Reading: Chapters 8 & 9 1 Turing Machines (TM) Generalize the class of CFLs: Recursively Enumerable Languages Recursive Languages Context-Free Languages Regular Languages

More information

CISC 876: Kolmogorov Complexity

CISC 876: Kolmogorov Complexity March 27, 2007 Outline 1 Introduction 2 Definition Incompressibility and Randomness 3 Prefix Complexity Resource-Bounded K-Complexity 4 Incompressibility Method Gödel s Incompleteness Theorem 5 Outline

More information

Theory of Computation

Theory of Computation Theory of Computation Dr. Sarmad Abbasi Dr. Sarmad Abbasi () Theory of Computation 1 / 33 Lecture 20: Overview Incompressible strings Minimal Length Descriptions Descriptive Complexity Dr. Sarmad Abbasi

More information

Lecture Notes: The Halting Problem; Reductions

Lecture Notes: The Halting Problem; Reductions Lecture Notes: The Halting Problem; Reductions COMS W3261 Columbia University 20 Mar 2012 1 Review Key point. Turing machines can be encoded as strings, and other Turing machines can read those strings

More information

Lecture 16: Time Complexity and P vs NP

Lecture 16: Time Complexity and P vs NP 6.045 Lecture 16: Time Complexity and P vs NP 1 Time-Bounded Complexity Classes Definition: TIME(t(n)) = { L there is a Turing machine M with time complexity O(t(n)) so that L = L(M) } = { L L is a language

More information

Peano Arithmetic. CSC 438F/2404F Notes (S. Cook) Fall, Goals Now

Peano Arithmetic. CSC 438F/2404F Notes (S. Cook) Fall, Goals Now CSC 438F/2404F Notes (S. Cook) Fall, 2008 Peano Arithmetic Goals Now 1) We will introduce a standard set of axioms for the language L A. The theory generated by these axioms is denoted PA and called Peano

More information

Lecture 23: Rice Theorem and Turing machine behavior properties 21 April 2009

Lecture 23: Rice Theorem and Turing machine behavior properties 21 April 2009 CS 373: Theory of Computation Sariel Har-Peled and Madhusudan Parthasarathy Lecture 23: Rice Theorem and Turing machine behavior properties 21 April 2009 This lecture covers Rice s theorem, as well as

More information

Computability Theory. CS215, Lecture 6,

Computability Theory. CS215, Lecture 6, Computability Theory CS215, Lecture 6, 2000 1 The Birth of Turing Machines At the end of the 19th century, Gottlob Frege conjectured that mathematics could be built from fundamental logic In 1900 David

More information

The Turing machine model of computation

The Turing machine model of computation The Turing machine model of computation For most of the remainder of the course we will study the Turing machine model of computation, named after Alan Turing (1912 1954) who proposed the model in 1936.

More information

Kolmogorov complexity and its applications

Kolmogorov complexity and its applications Spring, 2009 Kolmogorov complexity and its applications Paul Vitanyi Computer Science University of Amsterdam http://www.cwi.nl/~paulv/course-kc We live in an information society. Information science is

More information

Chomsky Normal Form and TURING MACHINES. TUESDAY Feb 4

Chomsky Normal Form and TURING MACHINES. TUESDAY Feb 4 Chomsky Normal Form and TURING MACHINES TUESDAY Feb 4 CHOMSKY NORMAL FORM A context-free grammar is in Chomsky normal form if every rule is of the form: A BC A a S ε B and C aren t start variables a is

More information

Kolmogorov complexity and its applications

Kolmogorov complexity and its applications CS860, Winter, 2010 Kolmogorov complexity and its applications Ming Li School of Computer Science University of Waterloo http://www.cs.uwaterloo.ca/~mli/cs860.html We live in an information society. Information

More information

7.1 The Origin of Computer Science

7.1 The Origin of Computer Science CS125 Lecture 7 Fall 2016 7.1 The Origin of Computer Science Alan Mathison Turing (1912 1954) turing.jpg 170!201 pixels On Computable Numbers, with an Application to the Entscheidungsproblem 1936 1936:

More information

FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY

FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY 15-453 FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY Chomsky Normal Form and TURING MACHINES TUESDAY Feb 4 CHOMSKY NORMAL FORM A context-free grammar is in Chomsky normal form if every rule is of the form:

More information

COS597D: Information Theory in Computer Science October 19, Lecture 10

COS597D: Information Theory in Computer Science October 19, Lecture 10 COS597D: Information Theory in Computer Science October 9, 20 Lecture 0 Lecturer: Mark Braverman Scribe: Andrej Risteski Kolmogorov Complexity In the previous lectures, we became acquainted with the concept

More information

An example of a decidable language that is not a CFL Implementation-level description of a TM State diagram of TM

An example of a decidable language that is not a CFL Implementation-level description of a TM State diagram of TM Turing Machines Review An example of a decidable language that is not a CFL Implementation-level description of a TM State diagram of TM Varieties of TMs Multi-Tape TMs Nondeterministic TMs String Enumerators

More information

Institute for Applied Information Processing and Communications (IAIK) Secure & Correct Systems. Decidability

Institute for Applied Information Processing and Communications (IAIK) Secure & Correct Systems. Decidability Decidability and the Undecidability of Predicate Logic IAIK Graz University of Technology georg.hofferek@iaik.tugraz.at 1 Fork of ways Brainteaser: Labyrinth Guards One to salvation One to perdition Two

More information

DRAFT. Diagonalization. Chapter 4

DRAFT. Diagonalization. Chapter 4 Chapter 4 Diagonalization..the relativized P =?NP question has a positive answer for some oracles and a negative answer for other oracles. We feel that this is further evidence of the difficulty of the

More information

Lecture 22: PSPACE

Lecture 22: PSPACE 6.045 Lecture 22: PSPACE 1 VOTE VOTE VOTE For your favorite course on automata and complexity Please complete the online subject evaluation for 6.045 2 Final Exam Information Who: You On What: Everything

More information

CSCE 551: Chin-Tser Huang. University of South Carolina

CSCE 551: Chin-Tser Huang. University of South Carolina CSCE 551: Theory of Computation Chin-Tser Huang huangct@cse.sc.edu University of South Carolina Church-Turing Thesis The definition of the algorithm came in the 1936 papers of Alonzo Church h and Alan

More information

Further discussion of Turing machines

Further discussion of Turing machines Further discussion of Turing machines In this lecture we will discuss various aspects of decidable and Turing-recognizable languages that were not mentioned in previous lectures. In particular, we will

More information

Recovery Based on Kolmogorov Complexity in Underdetermined Systems of Linear Equations

Recovery Based on Kolmogorov Complexity in Underdetermined Systems of Linear Equations Recovery Based on Kolmogorov Complexity in Underdetermined Systems of Linear Equations David Donoho Department of Statistics Stanford University Email: donoho@stanfordedu Hossein Kakavand, James Mammen

More information

(a) Definition of TMs. First Problem of URMs

(a) Definition of TMs. First Problem of URMs Sec. 4: Turing Machines First Problem of URMs (a) Definition of the Turing Machine. (b) URM computable functions are Turing computable. (c) Undecidability of the Turing Halting Problem That incrementing

More information

Introduction to Kolmogorov Complexity

Introduction to Kolmogorov Complexity Introduction to Kolmogorov Complexity Marcus Hutter Canberra, ACT, 0200, Australia http://www.hutter1.net/ ANU Marcus Hutter - 2 - Introduction to Kolmogorov Complexity Abstract In this talk I will give

More information

A version of for which ZFC can not predict a single bit Robert M. Solovay May 16, Introduction In [2], Chaitin introd

A version of for which ZFC can not predict a single bit Robert M. Solovay May 16, Introduction In [2], Chaitin introd CDMTCS Research Report Series A Version of for which ZFC can not Predict a Single Bit Robert M. Solovay University of California at Berkeley CDMTCS-104 May 1999 Centre for Discrete Mathematics and Theoretical

More information

Computational Complexity: A Modern Approach. Draft of a book: Dated January 2007 Comments welcome!

Computational Complexity: A Modern Approach. Draft of a book: Dated January 2007 Comments welcome! i Computational Complexity: A Modern Approach Draft of a book: Dated January 2007 Comments welcome! Sanjeev Arora and Boaz Barak Princeton University complexitybook@gmail.com Not to be reproduced or distributed

More information

Nondeterministic finite automata

Nondeterministic finite automata Lecture 3 Nondeterministic finite automata This lecture is focused on the nondeterministic finite automata (NFA) model and its relationship to the DFA model. Nondeterminism is an important concept in the

More information

Theory of Computation

Theory of Computation Theory of Computation Dr. Sarmad Abbasi Dr. Sarmad Abbasi () Theory of Computation / Lecture 3: Overview Decidability of Logical Theories Presburger arithmetic Decidability of Presburger Arithmetic Dr.

More information

Nondeterministic Finite Automata

Nondeterministic Finite Automata Nondeterministic Finite Automata Mahesh Viswanathan Introducing Nondeterminism Consider the machine shown in Figure. Like a DFA it has finitely many states and transitions labeled by symbols from an input

More information

Homework 8. a b b a b a b. two-way, read/write

Homework 8. a b b a b a b. two-way, read/write Homework 8 309 Homework 8 1. Describe a TM that accepts the set {a n n is a power of 2}. Your description should be at the level of the descriptions in Lecture 29 of the TM that accepts {ww w Σ } and the

More information

Computability Theory

Computability Theory Computability Theory Cristian S. Calude and Nicholas J. Hay May June 2009 Computability Theory 1 / 155 1 Register machines 2 Church-Turing thesis 3 Decidability 4 Reducibility 5 A definition of information

More information

Computability and Complexity Theory

Computability and Complexity Theory Discrete Math for Bioinformatics WS 09/10:, by A Bockmayr/K Reinert, January 27, 2010, 18:39 9001 Computability and Complexity Theory Computability and complexity Computability theory What problems can

More information

Definition: Alternating time and space Game Semantics: State of machine determines who

Definition: Alternating time and space Game Semantics: State of machine determines who CMPSCI 601: Recall From Last Time Lecture Definition: Alternating time and space Game Semantics: State of machine determines who controls, White wants it to accept, Black wants it to reject. White wins

More information

Turing Machines Part II

Turing Machines Part II Turing Machines Part II Hello Hello Condensed Slide Slide Readers! Readers! This This lecture lecture is is almost almost entirely entirely animations that that show show how how each each Turing Turing

More information

Primitive recursive functions: decidability problems

Primitive recursive functions: decidability problems Primitive recursive functions: decidability problems Armando B. Matos October 24, 2014 Abstract Although every primitive recursive (PR) function is total, many problems related to PR functions are undecidable.

More information

Theory of Computation

Theory of Computation Theory of Computation Lecture #10 Sarmad Abbasi Virtual University Sarmad Abbasi (Virtual University) Theory of Computation 1 / 43 Lecture 10: Overview Linear Bounded Automata Acceptance Problem for LBAs

More information

Busch Complexity Lectures: Turing s Thesis. Costas Busch - LSU 1

Busch Complexity Lectures: Turing s Thesis. Costas Busch - LSU 1 Busch Complexity Lectures: Turing s Thesis Costas Busch - LSU 1 Turing s thesis (1930): Any computation carried out by mechanical means can be performed by a Turing Machine Costas Busch - LSU 2 Algorithm:

More information

CSCI3390-Assignment 2 Solutions

CSCI3390-Assignment 2 Solutions CSCI3390-Assignment 2 Solutions due February 3, 2016 1 TMs for Deciding Languages Write the specification of a Turing machine recognizing one of the following three languages. Do one of these problems.

More information

Notes on Complexity Theory Last updated: October, Lecture 6

Notes on Complexity Theory Last updated: October, Lecture 6 Notes on Complexity Theory Last updated: October, 2015 Lecture 6 Notes by Jonathan Katz, lightly edited by Dov Gordon 1 PSPACE and PSPACE-Completeness As in our previous study of N P, it is useful to identify

More information

The axiomatic power of Kolmogorov complexity

The axiomatic power of Kolmogorov complexity The axiomatic power of Kolmogorov complexity Laurent Bienvenu 1, Andrei Romashchenko 2, Alexander Shen 2, Antoine Taveneaux 1, and Stijn Vermeeren 3 1 LIAFA, CNRS & Université Paris 7 2 LIRMM, CNRS & Université

More information

Stochasticity in Algorithmic Statistics for Polynomial Time

Stochasticity in Algorithmic Statistics for Polynomial Time Stochasticity in Algorithmic Statistics for Polynomial Time Alexey Milovanov 1 and Nikolay Vereshchagin 2 1 National Research University Higher School of Economics, Moscow, Russia almas239@gmail.com 2

More information

Notes on State Minimization

Notes on State Minimization U.C. Berkeley CS172: Automata, Computability and Complexity Handout 1 Professor Luca Trevisan 2/3/2015 Notes on State Minimization These notes present a technique to prove a lower bound on the number of

More information

Chapter 2 Algorithms and Computation

Chapter 2 Algorithms and Computation Chapter 2 Algorithms and Computation In this chapter, we first discuss the principles of algorithm and computation in general framework, common both in classical and quantum computers, then we go to the

More information

More About Turing Machines. Programming Tricks Restrictions Extensions Closure Properties

More About Turing Machines. Programming Tricks Restrictions Extensions Closure Properties More About Turing Machines Programming Tricks Restrictions Extensions Closure Properties 1 Overview At first, the TM doesn t look very powerful. Can it really do anything a computer can? We ll discuss

More information

A user-friendly Introduction to (un)computability and Unprovability via Church s Thesis

A user-friendly Introduction to (un)computability and Unprovability via Church s Thesis A user-friendly Introduction to (un)computability and Unprovability via Church s Thesis Computability is the part of logic that gives a mathematically precise formulation to the concepts algorithm, mechanical

More information

Lecture 12: Interactive Proofs

Lecture 12: Interactive Proofs princeton university cos 522: computational complexity Lecture 12: Interactive Proofs Lecturer: Sanjeev Arora Scribe:Carl Kingsford Recall the certificate definition of NP. We can think of this characterization

More information

What One Has to Know when Attacking P vs. NP. Juraj Hromkovič and Peter Rossmanith

What One Has to Know when Attacking P vs. NP. Juraj Hromkovič and Peter Rossmanith What One Has to Know when Attacking P vs. NP Juraj Hromkovič and Peter Rossmanith September 12, 2017 FCT 2017 [Chaitin, JACM 1974] There exists d N such that, for all n d and all x {0, 1}, there does not

More information

On Rice s theorem. Hans Hüttel. October 2001

On Rice s theorem. Hans Hüttel. October 2001 On Rice s theorem Hans Hüttel October 2001 We have seen that there exist languages that are Turing-acceptable but not Turing-decidable. An important example of such a language was the language of the Halting

More information

Kolmogorov Complexity in Randomness Extraction

Kolmogorov Complexity in Randomness Extraction LIPIcs Leibniz International Proceedings in Informatics Kolmogorov Complexity in Randomness Extraction John M. Hitchcock, A. Pavan 2, N. V. Vinodchandran 3 Department of Computer Science, University of

More information

Turing machines COMS Ashley Montanaro 21 March Department of Computer Science, University of Bristol Bristol, UK

Turing machines COMS Ashley Montanaro 21 March Department of Computer Science, University of Bristol Bristol, UK COMS11700 Turing machines Department of Computer Science, University of Bristol Bristol, UK 21 March 2014 COMS11700: Turing machines Slide 1/15 Introduction We have seen two models of computation: finite

More information

KOLMOGOROV COMPLEXITY AND ALGORITHMIC RANDOMNESS

KOLMOGOROV COMPLEXITY AND ALGORITHMIC RANDOMNESS KOLMOGOROV COMPLEXITY AND ALGORITHMIC RANDOMNESS HENRY STEINITZ Abstract. This paper aims to provide a minimal introduction to algorithmic randomness. In particular, we cover the equivalent 1-randomness

More information

Classical Propositional Logic

Classical Propositional Logic The Language of A Henkin-style Proof for Natural Deduction January 16, 2013 The Language of A Henkin-style Proof for Natural Deduction Logic Logic is the science of inference. Given a body of information,

More information

The Polynomial Hierarchy

The Polynomial Hierarchy The Polynomial Hierarchy Slides based on S.Aurora, B.Barak. Complexity Theory: A Modern Approach. Ahto Buldas Ahto.Buldas@ut.ee Motivation..synthesizing circuits is exceedingly difficulty. It is even

More information

198:538 Lecture Given on February 23rd, 1998

198:538 Lecture Given on February 23rd, 1998 198:538 Lecture Given on February 23rd, 1998 Lecturer: Professor Eric Allender Scribe: Sunny Daniels November 25, 1998 1 A Result From theorem 5 of the lecture on 16th February 1998, together with the

More information

Semantics and Pragmatics of NLP

Semantics and Pragmatics of NLP Semantics and Pragmatics of NLP Alex Ewan School of Informatics University of Edinburgh 28 January 2008 1 2 3 Taking Stock We have: Introduced syntax and semantics for FOL plus lambdas. Represented FOL

More information

Solution to CS375 Homework Assignment 11 (40 points) Due date: 4/26/2017

Solution to CS375 Homework Assignment 11 (40 points) Due date: 4/26/2017 Solution to CS375 Homework Assignment 11 (40 points) Due date: 4/26/2017 1. Find a Greibach normal form for the following given grammar. (10 points) S bab A BAa a B bb Ʌ Solution: (1) Since S does not

More information

6.841/18.405J: Advanced Complexity Wednesday, February 12, Lecture Lecture 3

6.841/18.405J: Advanced Complexity Wednesday, February 12, Lecture Lecture 3 6.841/18.405J: Advanced Complexity Wednesday, February 12, 2003 Lecture Lecture 3 Instructor: Madhu Sudan Scribe: Bobby Kleinberg 1 The language MinDNF At the end of the last lecture, we introduced the

More information

First Order Logic: Syntax and Semantics

First Order Logic: Syntax and Semantics CS1081 First Order Logic: Syntax and Semantics COMP30412 Sean Bechhofer sean.bechhofer@manchester.ac.uk Problems Propositional logic isn t very expressive As an example, consider p = Scotland won on Saturday

More information

Sophistication Revisited

Sophistication Revisited Sophistication Revisited Luís Antunes Lance Fortnow August 30, 007 Abstract Kolmogorov complexity measures the ammount of information in a string as the size of the shortest program that computes the string.

More information

ECS 120: Automata Theory and Formal Languages. UC Davis Phillip Rogaway December 13, ECS 120 Final { Fall 1995

ECS 120: Automata Theory and Formal Languages. UC Davis Phillip Rogaway December 13, ECS 120 Final { Fall 1995 ECS 120: Automata Theory and Formal Languages Handout Z UC Davis Phillip Rogaway December 13, 1995 ECS 120 Final { Fall 1995 Instructions: Check that your exam has all 7 problems (pages 2{9). You'll also

More information

PS2 - Comments. University of Virginia - cs3102: Theory of Computation Spring 2010

PS2 - Comments. University of Virginia - cs3102: Theory of Computation Spring 2010 University of Virginia - cs3102: Theory of Computation Spring 2010 PS2 - Comments Average: 77.4 (full credit for each question is 100 points) Distribution (of 54 submissions): 90, 12; 80 89, 11; 70-79,

More information

An Optimal Lower Bound for Nonregular Languages

An Optimal Lower Bound for Nonregular Languages An Optimal Lower Bound for Nonregular Languages Alberto Bertoni Carlo Mereghetti Giovanni Pighizzini Dipartimento di Scienze dell Informazione Università degli Studi di Milano via Comelico, 39 2035 Milano

More information

3 Self-Delimiting Kolmogorov complexity

3 Self-Delimiting Kolmogorov complexity 3 Self-Delimiting Kolmogorov complexity 3. Prefix codes A set is called a prefix-free set (or a prefix set) if no string in the set is the proper prefix of another string in it. A prefix set cannot therefore

More information

Sample Project: Simulation of Turing Machines by Machines with only Two Tape Symbols

Sample Project: Simulation of Turing Machines by Machines with only Two Tape Symbols Sample Project: Simulation of Turing Machines by Machines with only Two Tape Symbols The purpose of this document is to illustrate what a completed project should look like. I have chosen a problem that

More information

Theory of Computation CS3102 Spring 2015 A tale of computers, math, problem solving, life, love and tragic death

Theory of Computation CS3102 Spring 2015 A tale of computers, math, problem solving, life, love and tragic death Theory of Computation CS3102 Spring 2015 A tale of computers, math, problem solving, life, love and tragic death Robbie Hott www.cs.virginia.edu/~jh2jf Department of Computer Science University of Virginia

More information

Kvantifikator för en Dag

Kvantifikator för en Dag Philosophical Communications, Web Series, No. 35, pp. 155-162 Dept. of Philosophy, Göteborg University, Sweden ISSN 1652-0459 Kvantifikator för en Dag Essays dedicated to Dag Westerståhl on his sixtieth

More information

Chapter 6: Turing Machines

Chapter 6: Turing Machines Chapter 6: Turing Machines 6.1 The Turing Machine Definition A deterministic Turing machine (DTM) M is specified by a sextuple (Q, Σ, Γ, δ, s, f), where Q is a finite set of states; Σ is an alphabet of

More information

which possibility holds in an ensemble with a given a priori probability p k log 2

which possibility holds in an ensemble with a given a priori probability p k log 2 ALGORITHMIC INFORMATION THEORY Encyclopedia of Statistical Sciences, Volume 1, Wiley, New York, 1982, pp. 38{41 The Shannon entropy* concept of classical information theory* [9] is an ensemble notion it

More information

Algorithmic Information Theory

Algorithmic Information Theory Algorithmic Information Theory Peter D. Grünwald CWI, P.O. Box 94079 NL-1090 GB Amsterdam, The Netherlands E-mail: pdg@cwi.nl Paul M.B. Vitányi CWI, P.O. Box 94079 NL-1090 GB Amsterdam The Netherlands

More information

FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY

FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY 5-453 FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY 5-453 FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY YOU NEED TO PICK UP THE SYLLABUS, THE COURSE SCHEDULE, THE PROJECT INFO SHEET, TODAY S CLASS NOTES

More information

E.g. The set RE of regular expressions is given by the following rules:

E.g. The set RE of regular expressions is given by the following rules: 1 Lecture Summary We gave a brief overview of inductively defined sets, inductively defined functions, and proofs by structural induction We showed how to prove that a machine is correct using induction

More information

ECS 120 Lesson 15 Turing Machines, Pt. 1

ECS 120 Lesson 15 Turing Machines, Pt. 1 ECS 120 Lesson 15 Turing Machines, Pt. 1 Oliver Kreylos Wednesday, May 2nd, 2001 Before we can start investigating the really interesting problems in theoretical computer science, we have to introduce

More information

Regular Languages and Finite Automata

Regular Languages and Finite Automata Regular Languages and Finite Automata Theorem: Every regular language is accepted by some finite automaton. Proof: We proceed by induction on the (length of/structure of) the description of the regular

More information

10.4 The Kruskal Katona theorem

10.4 The Kruskal Katona theorem 104 The Krusal Katona theorem 141 Example 1013 (Maximum weight traveling salesman problem We are given a complete directed graph with non-negative weights on edges, and we must find a maximum weight Hamiltonian

More information

Turing Machines. The Language Hierarchy. Context-Free Languages. Regular Languages. Courtesy Costas Busch - RPI 1

Turing Machines. The Language Hierarchy. Context-Free Languages. Regular Languages. Courtesy Costas Busch - RPI 1 Turing Machines a n b n c The anguage Hierarchy n? ww? Context-Free anguages a n b n egular anguages a * a *b* ww Courtesy Costas Busch - PI a n b n c n Turing Machines anguages accepted by Turing Machines

More information

Section 3.1: Direct Proof and Counterexample 1

Section 3.1: Direct Proof and Counterexample 1 Section 3.1: Direct Proof and Counterexample 1 In this chapter, we introduce the notion of proof in mathematics. A mathematical proof is valid logical argument in mathematics which shows that a given conclusion

More information

cse303 ELEMENTS OF THE THEORY OF COMPUTATION Professor Anita Wasilewska

cse303 ELEMENTS OF THE THEORY OF COMPUTATION Professor Anita Wasilewska cse303 ELEMENTS OF THE THEORY OF COMPUTATION Professor Anita Wasilewska LECTURE 14 SMALL REVIEW FOR FINAL SOME Y/N QUESTIONS Q1 Given Σ =, there is L over Σ Yes: = {e} and L = {e} Σ Q2 There are uncountably

More information

2.2 Inverses and GCDs

2.2 Inverses and GCDs 34 CHAPTER 2. CRYPTOGRAPHY AND NUMBER THEORY 2.2 Inverses and GCDs 2.2.1 Inverses mod p In the last section we explored the multiplication in Z n. We saw in the special case with n =12 and a = 4 that if

More information

2 P vs. NP and Diagonalization

2 P vs. NP and Diagonalization 2 P vs NP and Diagonalization CS 6810 Theory of Computing, Fall 2012 Instructor: David Steurer (sc2392) Date: 08/28/2012 In this lecture, we cover the following topics: 1 3SAT is NP hard; 2 Time hierarchies;

More information

6.045J/18.400J: Automata, Computability and Complexity Final Exam. There are two sheets of scratch paper at the end of this exam.

6.045J/18.400J: Automata, Computability and Complexity Final Exam. There are two sheets of scratch paper at the end of this exam. 6.045J/18.400J: Automata, Computability and Complexity May 20, 2005 6.045 Final Exam Prof. Nancy Lynch Name: Please write your name on each page. This exam is open book, open notes. There are two sheets

More information

FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY

FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY 15-453 FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY THE PUMPING LEMMA FOR REGULAR LANGUAGES and REGULAR EXPRESSIONS TUESDAY Jan 21 WHICH OF THESE ARE REGULAR? B = {0 n 1 n n 0} C = { w w has equal number

More information

2 Natural Proofs: a barrier for proving circuit lower bounds

2 Natural Proofs: a barrier for proving circuit lower bounds Topics in Theoretical Computer Science April 4, 2016 Lecturer: Ola Svensson Lecture 6 (Notes) Scribes: Ola Svensson Disclaimer: These notes were written for the lecturer only and may contain inconsistent

More information

Kolmogorov complexity ; induction, prediction and compression

Kolmogorov complexity ; induction, prediction and compression Kolmogorov complexity ; induction, prediction and compression Contents 1 Motivation for Kolmogorov complexity 1 2 Formal Definition 2 3 Trying to compute Kolmogorov complexity 3 4 Standard upper bounds

More information

Examples of Unlimited Register Machine programs for Turingcomputable

Examples of Unlimited Register Machine programs for Turingcomputable Examples of Unlimited Register Machine programs for Turingcomputable functions 18 February 2012 at 13:01 Public It will be a matter of interest for me over the next few months to formulate (and collect)

More information

One-to-one functions and onto functions

One-to-one functions and onto functions MA 3362 Lecture 7 - One-to-one and Onto Wednesday, October 22, 2008. Objectives: Formalize definitions of one-to-one and onto One-to-one functions and onto functions At the level of set theory, there are

More information

acs-07: Decidability Decidability Andreas Karwath und Malte Helmert Informatik Theorie II (A) WS2009/10

acs-07: Decidability Decidability Andreas Karwath und Malte Helmert Informatik Theorie II (A) WS2009/10 Decidability Andreas Karwath und Malte Helmert 1 Overview An investigation into the solvable/decidable Decidable languages The halting problem (undecidable) 2 Decidable problems? Acceptance problem : decide

More information

b + O(n d ) where a 1, b > 1, then O(n d log n) if a = b d d ) if a < b d O(n log b a ) if a > b d

b + O(n d ) where a 1, b > 1, then O(n d log n) if a = b d d ) if a < b d O(n log b a ) if a > b d CS161, Lecture 4 Median, Selection, and the Substitution Method Scribe: Albert Chen and Juliana Cook (2015), Sam Kim (2016), Gregory Valiant (2017) Date: January 23, 2017 1 Introduction Last lecture, we

More information

Lecture 14: IP = PSPACE

Lecture 14: IP = PSPACE IAS/PCMI Summer Session 2000 Clay Mathematics Undergraduate Program Basic Course on Computational Complexity Lecture 14: IP = PSPACE David Mix Barrington and Alexis Maciel August 3, 2000 1. Overview We

More information

Properties of Regular Languages. BBM Automata Theory and Formal Languages 1

Properties of Regular Languages. BBM Automata Theory and Formal Languages 1 Properties of Regular Languages BBM 401 - Automata Theory and Formal Languages 1 Properties of Regular Languages Pumping Lemma: Every regular language satisfies the pumping lemma. A non-regular language

More information

3 Propositional Logic

3 Propositional Logic 3 Propositional Logic 3.1 Syntax 3.2 Semantics 3.3 Equivalence and Normal Forms 3.4 Proof Procedures 3.5 Properties Propositional Logic (25th October 2007) 1 3.1 Syntax Definition 3.0 An alphabet Σ consists

More information

A.Antonopoulos 18/1/2010

A.Antonopoulos 18/1/2010 Class DP Basic orems 18/1/2010 Class DP Basic orems 1 Class DP 2 Basic orems Class DP Basic orems TSP Versions 1 TSP (D) 2 EXACT TSP 3 TSP COST 4 TSP (1) P (2) P (3) P (4) DP Class Class DP Basic orems

More information

CSCI FOUNDATIONS OF COMPUTER SCIENCE

CSCI FOUNDATIONS OF COMPUTER SCIENCE 1 CSCI- 2200 FOUNDATIONS OF COMPUTER SCIENCE Spring 2015 My 7, 2015 2 Announcements Homework 9 is due now. Some finl exm review problems will be posted on the web site tody. These re prcqce problems not

More information

LECTURE 1. Logic and Proofs

LECTURE 1. Logic and Proofs LECTURE 1 Logic and Proofs The primary purpose of this course is to introduce you, most of whom are mathematics majors, to the most fundamental skills of a mathematician; the ability to read, write, and

More information