# CISC 876: Kolmogorov Complexity

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 March 27, 2007

2 Outline 1 Introduction 2 Definition Incompressibility and Randomness 3 Prefix Complexity Resource-Bounded K-Complexity 4 Incompressibility Method Gödel s Incompleteness Theorem 5

3 Outline 1 Introduction 2 Definition Incompressibility and Randomness 3 Prefix Complexity Resource-Bounded K-Complexity 4 Incompressibility Method Gödel s Incompleteness Theorem 5

4 Complexity of Objects Example Which of these is more complex?

5 Complexity of Objects Example Which of these is more complex? Intuition The first has a simple description: print 1 16 times. There is no (obvious) description for the second string that is essentially shorter than listing its digits.

6 Complexity of Objects Example Which of these is more complex? Intuition The first has a simple description: print 1 16 times. There is no (obvious) description for the second string that is essentially shorter than listing its digits. Kolmogorov complexity formalizes this intuitive notion of complexity.

7 Complexity As Predictive Power Solomonoff s Idea Suppose a scientist takes a sequence of measurements: x = {0, 1}. The scientist would like to formulate a hypothesis that predicts the future content of the sequence. Among the infinite number of possible hypotheses, which should be preferred?

8 Complexity As Predictive Power Solomonoff s Idea Suppose a scientist takes a sequence of measurements: x = {0, 1}. The scientist would like to formulate a hypothesis that predicts the future content of the sequence. Among the infinite number of possible hypotheses, which should be preferred? Occam s Razor Choose the simplest hypothesis that is consistent with the data

9 Algorithmic Information Theory Algorithmic information theory is the result of putting Shannon s information theory and Turing s computability theory into a cocktail shaker and shaking vigorously. G. J. Chaitin

10 Algorithmic Information Theory Algorithmic information theory is the result of putting Shannon s information theory and Turing s computability theory into a cocktail shaker and shaking vigorously. G. J. Chaitin AIT is a subfield of both information theory and computer science (Almost) simultaneously and independently developed by 1962: introduced by R. J. Solomonoff as part of work on inductive inference 1963: A. N. Kolmogorov 1965: G. J. Chaitin (while an 18-year old undergraduate!) Also known as Kolmogorov-Chaitin complexity, descriptional complexity, program-size complexity,...

11 Outline Introduction Definition Incompressibility and Randomness 1 Introduction 2 Definition Incompressibility and Randomness 3 Prefix Complexity Resource-Bounded K-Complexity 4 Incompressibility Method Gödel s Incompleteness Theorem 5

12 Definition Introduction Definition Incompressibility and Randomness Definition The Kolmogorov complexity of a string x is the length of the smallest program that outputs x, relative to some model of computation. That is, for some computer f. C f (x) = min p { p : f (p) = x} Informally, C(x) measures the information content, degree of redundancy, degree of structure, of x

13 Definition Incompressibility and Randomness Universality Problem C f (x) depends on both f and x. Can we measure the inherent information in x, independent of the choice of f?

14 Definition Incompressibility and Randomness Universality Problem C f (x) depends on both f and x. Can we measure the inherent information in x, independent of the choice of f? Theorem (Invariance Theorem) There exists a universal description method ψ 0, such that: C ψ0 (x) C ψ (x) + c for some constant c that depends on ψ and ψ 0 (but not on x). Proof Idea. Follows from the existence of a universal Turing machine: accept a description of ψ and ψ s program for x

15 Definition Incompressibility and Randomness Implications Theorem For all universal description methods f, g: C f (x) C g (x) c for some constant c that depends only on f and g. This is crucial to the usefulness of the complexity measure The universal description method does not necessarily give the shortest description of each object, but no other description method can improve on it by more than an additive constant We typically write C(x) = C ψ0 (x), use Turing machines as ψ 0, and limit our analysis to within an additive constant

16 Definition Incompressibility and Randomness Conditional Complexity Definition The conditional Kolmogorov complexity of a string x, relative to a string y and a model of computation f, is: C f (x y) = min{ p : C f (p, y) = x} C f (x) = C f (x ɛ) C(x y) is the size of the minimal program for x when started with input y C(x : y) = C(x) C(x y) describes the information y contains about x When C(x : y) = C(x), x and y are algorithmically independent

17 Definition Incompressibility and Randomness Simple Results Upper Bound On C(x) There is a constant c, such that for all x: C(x) x + c (Proving a lower bound on C(x) is not as straightforward.) Structure and Complexity For each constant k, there is a constant c such that for all x: C(x k ) C(x) + c

18 Definition Incompressibility and Randomness Incompressibility and Randomness Definition A string x is incompressible if C(x) x Maximal information content, no redundancy: algorithmically random Short programs encode patterns in non-random strings Algorithmic randomness is not identical to the intuitive concept of randomness There is a short program for generating the digits of π, so they are highly non-random

19 Are There Incompressible Strings? Definition Incompressibility and Randomness Theorem For all n, there exists an incompressible string of length n Proof. There are 2 n strings of length n and fewer than 2 n descriptions that are shorter than n: n 1 2 i = 2 n 1 < 2 n i=0

20 Definition Incompressibility and Randomness Incompressibility Theorem We can extend the previous counting argument to show that the vast majority of strings are mostly incompressible Definition A string x is c-incompressible if C(x) x c, for some constant c. Theorem The number of strings of length n that are c-incompressible is at least 2 n 2 n c+1 + 1

21 Example Introduction Definition Incompressibility and Randomness For c = 10: The fraction of all strings of length n with complexity less than n 10 is smaller than: 2 n n =

22 Consequences Introduction Definition Incompressibility and Randomness Fact The probability that an infinite sequence obtained by independent tosses of a fair coin is algorithmically random is 1. Fact The minimal program for any string is algorithmically random.

23 Definition Incompressibility and Randomness Noncomputability Theorem Theorem C(x) is not a computable function. Proof. Will be presented shortly.

24 Definition Incompressibility and Randomness Conclusions Given any concrete string, we cannot show that it is random Apparent randomness may be the result of a hidden structure Wolfram s conjecture: much/all apparent physical randomness is ultimately the result of structure Almost all strings are algorithmically random, but we cannot exhibit any particular string that is random There are relatively few short programs, and relatively few objects of low complexity

25 Outline Introduction Prefix Complexity Resource-Bounded K-Complexity 1 Introduction 2 Definition Incompressibility and Randomness 3 Prefix Complexity Resource-Bounded K-Complexity 4 Incompressibility Method Gödel s Incompleteness Theorem 5

26 Prefix Complexity Resource-Bounded K-Complexity Additive Complexity Theorem Proof Idea. C(x, y) = C(x) + C(y) + O(log(min(C(x), C(y)))) 1 ( ): Construct a TM that accepts descriptions (programs) for x, y, and a way to distinguish them The length of the shorter input 2 ( ): It can be shown that we cannot do better than this for all but finitely many x, y

27 Prefix Complexity Resource-Bounded K-Complexity Consequences This is unfortunate: we would like K-complexity to be subadditive C(x) + C(y) should bound C(x, y) from above We would also like to combine subprograms by simple concatenation

28 Self-Delimiting Strings Prefix Complexity Resource-Bounded K-Complexity Definition A string is self-delimiting if it contains its own length.

29 Prefix Complexity Resource-Bounded K-Complexity Self-Delimiting Strings Definition A string is self-delimiting if it contains its own length. Procedure Prepend the string s length to the string Problem: how can we distinguish the end of the length from the start of the string itself?

30 Prefix Complexity Resource-Bounded K-Complexity Self-Delimiting Strings Definition A string is self-delimiting if it contains its own length. Procedure Prepend the string s length to the string Problem: how can we distinguish the end of the length from the start of the string itself? Solution: duplicate every bit of the length, then mark the end of the length with 01 or 10 A binary string of length n can be encoded in self-delimiting form in n + 2 log n bits

31 Prefix Complexity Resource-Bounded K-Complexity Prefix Complexity Most modern work on Kolmogorov complexity actually uses prefix complexity, a variant formulated by L. A. Levin (1974) K(x) is the size of the minimal self-delimiting program that outputs x; K(x) is subadditive No self-delimiting string is the prefix of another Various other helpful theoretical properties

32 of Results Introduction Prefix Complexity Resource-Bounded K-Complexity upper bounds: K(x) l(x) + 2 log l(x), K(x l(x)) = l(x)

33 Prefix Complexity Resource-Bounded K-Complexity of Results upper bounds: K(x) l(x) + 2 log l(x), K(x l(x)) = l(x) extra information: K(x y) K(x) K(x, y)

34 Prefix Complexity Resource-Bounded K-Complexity of Results upper bounds: K(x) l(x) + 2 log l(x), K(x l(x)) = l(x) extra information: K(x y) K(x) K(x, y) subadditive: K(x, y) K(x) + K(y)

35 Prefix Complexity Resource-Bounded K-Complexity of Results upper bounds: K(x) l(x) + 2 log l(x), K(x l(x)) = l(x) extra information: K(x y) K(x) K(x, y) subadditive: K(x, y) K(x) + K(y) symmetry of information: K(x, y) K(y, x)

36 Prefix Complexity Resource-Bounded K-Complexity of Results upper bounds: K(x) l(x) + 2 log l(x), K(x l(x)) = l(x) extra information: K(x y) K(x) K(x, y) subadditive: K(x, y) K(x) + K(y) symmetry of information: K(x, y) K(y, x) lower bound: K(x) l(x) for almost all x

37 Prefix Complexity Resource-Bounded K-Complexity Resource-Bounded Kolmogorov Complexity Definition Intuitively, a string has high logical depth if it is superficially random, but subtly redundant : the string has low complexity, but only for a computational model with access to a lot of resources

38 Prefix Complexity Resource-Bounded K-Complexity Resource-Bounded Kolmogorov Complexity Definition Intuitively, a string has high logical depth if it is superficially random, but subtly redundant : the string has low complexity, but only for a computational model with access to a lot of resources We can consider the complexity of a string, relative to a computational model with bounded space or time resources Typically harder to prove results than with unbounded K-complexity

39 Prefix Complexity Resource-Bounded K-Complexity Invariance Theorem with Resource Bounds Theorem (Invariance Theorem) There exists a universal description method ψ 0, such that for all other description methods ψ we have a constant c such that: C ct log n,cs ψ 0 (x) = C t,s ψ (x) + c Problem Considerably weaker Invariance Theorem: multiplicative constant factor in space complexity, multiplicative logarithmic factor in time complexity.

40 Relation To Other Fields Prefix Complexity Resource-Bounded K-Complexity Shannon s information theory The information required to select an element from a previously agreed-upon set of alternatives

41 Relation To Other Fields Prefix Complexity Resource-Bounded K-Complexity Shannon s information theory The information required to select an element from a previously agreed-upon set of alternatives Minimum Description Length (MDL) Place limitations on the computation model so the MDL of a string is computable Closer to learning theory and Solomonoff s work on inductive inference

42 Relation To Other Fields Prefix Complexity Resource-Bounded K-Complexity Shannon s information theory The information required to select an element from a previously agreed-upon set of alternatives Minimum Description Length (MDL) Place limitations on the computation model so the MDL of a string is computable Closer to learning theory and Solomonoff s work on inductive inference Circuit complexity Kolmogorov complexity considers Turing machines rather than Boolean circuits

43 Outline Introduction Incompressibility Method Gödel s Incompleteness Theorem 1 Introduction 2 Definition Incompressibility and Randomness 3 Prefix Complexity Resource-Bounded K-Complexity 4 Incompressibility Method Gödel s Incompleteness Theorem 5

44 Incompressibility Method Gödel s Incompleteness Theorem Incompressibility Method A general-purpose method for formal proofs; often an alternative to counting arguments or probabilistic arguments Typical Proof Structure. To show that almost all the objects in a given class have a certain property: 1 Choose a random object from the class 2 This object is incompressible, with probability 1 3 Prove that the property holds for the object 1 Assume that the property does not hold 2 Show that we can use the property to compress the object, yielding a contradiction

45 Incompressibility Method Gödel s Incompleteness Theorem Simple Example Theorem L = {0 k 1 k : k 1} is not regular. Proof Idea. 1 Choose k such that k is Kolmogorov-random

46 Incompressibility Method Gödel s Incompleteness Theorem Simple Example Theorem L = {0 k 1 k : k 1} is not regular. Proof Idea. 1 Choose k such that k is Kolmogorov-random 2 Assume that 0 k 1 k is a regular language, and is accepted by some finite automaton A

47 Incompressibility Method Gödel s Incompleteness Theorem Simple Example Theorem L = {0 k 1 k : k 1} is not regular. Proof Idea. 1 Choose k such that k is Kolmogorov-random 2 Assume that 0 k 1 k is a regular language, and is accepted by some finite automaton A 3 After input 0 k, A is in state q

48 Incompressibility Method Gödel s Incompleteness Theorem Simple Example Theorem L = {0 k 1 k : k 1} is not regular. Proof Idea. 1 Choose k such that k is Kolmogorov-random 2 Assume that 0 k 1 k is a regular language, and is accepted by some finite automaton A 3 After input 0 k, A is in state q 4 A and q form a concise description of k: running A from state q accepts only on an input of k consecutive 1s

49 Incompressibility Method Gödel s Incompleteness Theorem Simple Example Theorem L = {0 k 1 k : k 1} is not regular. Proof Idea. 1 Choose k such that k is Kolmogorov-random 2 Assume that 0 k 1 k is a regular language, and is accepted by some finite automaton A 3 After input 0 k, A is in state q 4 A and q form a concise description of k: running A from state q accepts only on an input of k consecutive 1s 5 This contradicts the assumption that k is incompressible

50 Properties of Formal Languages Incompressibility Method Gödel s Incompleteness Theorem Pumping lemmas are the standard tool for showing that a language is not in REG, DCFL, CFL,... Kolmogorov complexity provides an alternative way to characterize membership in these classes Can prove both regularity and non-regularity

51 Incompressibility Method Gödel s Incompleteness Theorem K-Complexity Analog to the Pumping Lemma for REG Lemma (Kolmogorov-Complexity-Regularity (KCR)) Let L be a regular language. Then for some c depending only on L and for each x, if y is the nth string in lexicographical order L x = {y : xy L}, then K(y) K(n) + c.

52 Incompressibility Method Gödel s Incompleteness Theorem K-Complexity Analog to the Pumping Lemma for REG Lemma (Kolmogorov-Complexity-Regularity (KCR)) Let L be a regular language. Then for some c depending only on L and for each x, if y is the nth string in lexicographical order L x = {y : xy L}, then K(y) K(n) + c. Proof. Any string y such that xy L, can be described by: 1 the description of the FA that accepts L 2 the state of the FA after processing x 3 the number n

53 Incompressibility Method Gödel s Incompleteness Theorem Example of KCR Lemma Fact L = {0 n : n is prime } is not regular.

54 Incompressibility Method Gödel s Incompleteness Theorem Example of KCR Lemma Fact L = {0 n : n is prime } is not regular. Proof. Assume that L is regular. Set xy = 0 p and x = 0 p, where p is the k th prime and p is the (k 1)th prime. It follows that y = 0 p p, n = 1, and K(p p ) = O(1).

55 Incompressibility Method Gödel s Incompleteness Theorem Example of KCR Lemma Fact L = {0 n : n is prime } is not regular. Proof. Assume that L is regular. Set xy = 0 p and x = 0 p, where p is the k th prime and p is the (k 1)th prime. It follows that y = 0 p p, n = 1, and K(p p ) = O(1). This is a contradiction: the difference between consecutive primes rises unbounded, so there are an unbounded number of integers with O(1) descriptions.

56 Incompressibility Method Gödel s Incompleteness Theorem Other of the Incompressibility Method Average-case complexity analysis Avoids the need to explicitly model the probability distribution of inputs E.g. heapsort Lower bounds analysis for problems Properties of random graphs Typically yields proofs that are shorter and more elegant than alternative techniques

57 Incompressibility Method Gödel s Incompleteness Theorem Historical Context: Formal Axiomatic Systems Turn of the 20th century: what constitutes a valid proof? David Hilbert s program: can we formalize mathematics? Hilbert s 2nd Problem (1900) Construct a single formal axiomatic system that contains all true arithmetical statements over the natural numbers: A finite number of axioms, and a deterministic inference procedure Consistent: no contradictions can be derived from the axioms Complete: all true statements can be derived from the axioms

58 Incompressibility Method Gödel s Incompleteness Theorem Gödel s Incompleteness Theorems Theorem (1st Incompleteness Theorem) Any computably enumerable, consistent formal axiomatic system containing elementary arithmetic is incomplete: there exist true, but unprovable (within the system) statements.

59 Incompressibility Method Gödel s Incompleteness Theorem Gödel s Incompleteness Theorems Theorem (1st Incompleteness Theorem) Any computably enumerable, consistent formal axiomatic system containing elementary arithmetic is incomplete: there exist true, but unprovable (within the system) statements. Theorem (2nd Incompleteness Theorem) The consistency of a formal axiomatic system that contains arithmetic cannot be proven within the system.

60 Consequences Introduction Incompressibility Method Gödel s Incompleteness Theorem For each true, unprovable statement, we can solve the problem by adding a new axiom to the system There are an infinity of such unprovable statements, so we never achieve completeness

61 Incompressibility Method Gödel s Incompleteness Theorem Consequences For each true, unprovable statement, we can solve the problem by adding a new axiom to the system There are an infinity of such unprovable statements, so we never achieve completeness Hilbert s program is not achievable: any single axiomatization of number theory cannot capture all number-theoretical truths However, does not invalidate formalism itself: many formal models are now necessary rather than a single one

62 Incompressibility Method Gödel s Incompleteness Theorem Consequences For each true, unprovable statement, we can solve the problem by adding a new axiom to the system There are an infinity of such unprovable statements, so we never achieve completeness Hilbert s program is not achievable: any single axiomatization of number theory cannot capture all number-theoretical truths However, does not invalidate formalism itself: many formal models are now necessary rather than a single one Provability is a weaker notion than truth. Douglas Hofstadter... and much more philosophical speculation in the same vein

63 Incompressibility Method Gödel s Incompleteness Theorem Connection to Kolmogorov Complexity Gödel s proof relies on an ingenious technique: in any formal system that contains arithmetic, we can construct a true theorem in the formal system that encodes the assertion This theorem is not provable within the system Neat, but an artificial construction How widespread are these true, unprovable statements? K-complexity allows a simple proof of Gödel s incompleteness results that sheds more light on the power of formal systems

64 Complexity of a Formal System Incompressibility Method Gödel s Incompleteness Theorem Definition The complexity of a formal system is the size of the minimal program that lists all the theorems in the system. Equivalently, a formal system s complexity is the size of the minimal encoding of the alphabet, axioms, and inference procedure

65 Incompressibility Method Gödel s Incompleteness Theorem Solving the Halting Problem Theorem A formal system of complexity n can solve the Halting Problem for programs smaller than n bits. Proof. The system can contain at most n bits of axioms. This is enough space to specify the number of n-bit programs that halt, or equivalently to identify the halting program with the longest runtime. Corollary A finite formal axiomatic system can only prove finitely many statements of the form C(x) > m.

66 Incompressibility Method Gödel s Incompleteness Theorem Corollary: K(x) Is Uncomputable Theorem No formal system of complexity n can prove that an object x has K(x) > n. Proof Idea. If the formal system can prove that x has complexity n = K(x), this encodes a description of x in n bits. Since n > n, we have a contradiction.

67 AIT Restatement of Incompleteness Incompressibility Method Gödel s Incompleteness Theorem Theorem There are true but unprovable statements in any consistent formal axiomatic system of finite size. Proof Idea. Follows from the earlier two results.

68 Outline 1 Introduction 2 Definition Incompressibility and Randomness 3 Prefix Complexity Resource-Bounded K-Complexity 4 Incompressibility Method Gödel s Incompleteness Theorem 5

69 Kolmogorov complexity measures the absolute information content of a string, to within an additive constant The uncomputability of K-complexity is an obstacle The incompressibility method is a useful (advanced) proof technique AIT allows a simple proof of the Incompleteness theorem, as well as more insight into the nature of formal axiomatic systems

### Computability Theory

Computability Theory Cristian S. Calude May 2012 Computability Theory 1 / 1 Bibliography M. Sipser. Introduction to the Theory of Computation, PWS 1997. (textbook) Computability Theory 2 / 1 Supplementary

### Kolmogorov complexity and its applications

Spring, 2009 Kolmogorov complexity and its applications Paul Vitanyi Computer Science University of Amsterdam http://www.cwi.nl/~paulv/course-kc We live in an information society. Information science is

### 3 Self-Delimiting Kolmogorov complexity

3 Self-Delimiting Kolmogorov complexity 3. Prefix codes A set is called a prefix-free set (or a prefix set) if no string in the set is the proper prefix of another string in it. A prefix set cannot therefore

### CS154, Lecture 12: Kolmogorov Complexity: A Universal Theory of Data Compression

CS154, Lecture 12: Kolmogorov Complexity: A Universal Theory of Data Compression Rosencrantz & Guildenstern Are Dead (Tom Stoppard) Rigged Lottery? And the winning numbers are: 1, 2, 3, 4, 5, 6 But is

### Kolmogorov complexity and its applications

CS860, Winter, 2010 Kolmogorov complexity and its applications Ming Li School of Computer Science University of Waterloo http://www.cs.uwaterloo.ca/~mli/cs860.html We live in an information society. Information

### Computability Theory. CS215, Lecture 6,

Computability Theory CS215, Lecture 6, 2000 1 The Birth of Turing Machines At the end of the 19th century, Gottlob Frege conjectured that mathematics could be built from fundamental logic In 1900 David

### Lecture 13: Foundations of Math and Kolmogorov Complexity

6.045 Lecture 13: Foundations of Math and Kolmogorov Complexity 1 Self-Reference and the Recursion Theorem 2 Lemma: There is a computable function q : Σ* Σ* such that for every string w, q(w) is the description

### FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY

15-453 FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY KOLMOGOROV-CHAITIN (descriptive) COMPLEXITY TUESDAY, MAR 18 CAN WE QUANTIFY HOW MUCH INFORMATION IS IN A STRING? A = 01010101010101010101010101010101

### ,

Kolmogorov Complexity Carleton College, CS 254, Fall 2013, Prof. Joshua R. Davis based on Sipser, Introduction to the Theory of Computation 1. Introduction Kolmogorov complexity is a theory of lossless

### Algorithmic Probability

Algorithmic Probability From Scholarpedia From Scholarpedia, the free peer-reviewed encyclopedia p.19046 Curator: Marcus Hutter, Australian National University Curator: Shane Legg, Dalle Molle Institute

### Algorithmic Information Theory

Algorithmic Information Theory Peter D. Grünwald CWI, P.O. Box 94079 NL-1090 GB Amsterdam, The Netherlands E-mail: pdg@cwi.nl Paul M.B. Vitányi CWI, P.O. Box 94079 NL-1090 GB Amsterdam The Netherlands

### Sophistication Revisited

Sophistication Revisited Luís Antunes Lance Fortnow August 30, 007 Abstract Kolmogorov complexity measures the ammount of information in a string as the size of the shortest program that computes the string.

### The axiomatic power of Kolmogorov complexity

The axiomatic power of Kolmogorov complexity Laurent Bienvenu 1, Andrei Romashchenko 2, Alexander Shen 2, Antoine Taveneaux 1, and Stijn Vermeeren 3 1 LIAFA, CNRS & Université Paris 7 2 LIRMM, CNRS & Université

### Introduction to Kolmogorov Complexity

Introduction to Kolmogorov Complexity Marcus Hutter Canberra, ACT, 0200, Australia http://www.hutter1.net/ ANU Marcus Hutter - 2 - Introduction to Kolmogorov Complexity Abstract In this talk I will give

### KOLMOGOROV COMPLEXITY AND ALGORITHMIC RANDOMNESS

KOLMOGOROV COMPLEXITY AND ALGORITHMIC RANDOMNESS HENRY STEINITZ Abstract. This paper aims to provide a minimal introduction to algorithmic randomness. In particular, we cover the equivalent 1-randomness

### COS597D: Information Theory in Computer Science October 19, Lecture 10

COS597D: Information Theory in Computer Science October 9, 20 Lecture 0 Lecturer: Mark Braverman Scribe: Andrej Risteski Kolmogorov Complexity In the previous lectures, we became acquainted with the concept

### Peano Arithmetic. CSC 438F/2404F Notes (S. Cook) Fall, Goals Now

CSC 438F/2404F Notes (S. Cook) Fall, 2008 Peano Arithmetic Goals Now 1) We will introduce a standard set of axioms for the language L A. The theory generated by these axioms is denoted PA and called Peano

### Recovery Based on Kolmogorov Complexity in Underdetermined Systems of Linear Equations

Recovery Based on Kolmogorov Complexity in Underdetermined Systems of Linear Equations David Donoho Department of Statistics Stanford University Email: donoho@stanfordedu Hossein Kakavand, James Mammen

### Kolmogorov Complexity

Kolmogorov Complexity Davide Basilio Bartolini University of Illinois at Chicago Politecnico di Milano dbarto3@uic.edu davide.bartolini@mail.polimi.it 1 Abstract What follows is a survey of the field of

### Stochasticity in Algorithmic Statistics for Polynomial Time

Stochasticity in Algorithmic Statistics for Polynomial Time Alexey Milovanov 1 and Nikolay Vereshchagin 2 1 National Research University Higher School of Economics, Moscow, Russia almas239@gmail.com 2

### Decision Problems with TM s. Lecture 31: Halting Problem. Universe of discourse. Semi-decidable. Look at following sets: CSCI 81 Spring, 2012

Decision Problems with TM s Look at following sets: Lecture 31: Halting Problem CSCI 81 Spring, 2012 Kim Bruce A TM = { M,w M is a TM and w L(M)} H TM = { M,w M is a TM which halts on input w} TOTAL TM

### DRAFT. Diagonalization. Chapter 4

Chapter 4 Diagonalization..the relativized P =?NP question has a positive answer for some oracles and a negative answer for other oracles. We feel that this is further evidence of the difficulty of the

### Finish K-Complexity, Start Time Complexity

6.045 Finish K-Complexity, Start Time Complexity 1 Kolmogorov Complexity Definition: The shortest description of x, denoted as d(x), is the lexicographically shortest string such that M(w) halts

### A version of for which ZFC can not predict a single bit Robert M. Solovay May 16, Introduction In [2], Chaitin introd

CDMTCS Research Report Series A Version of for which ZFC can not Predict a Single Bit Robert M. Solovay University of California at Berkeley CDMTCS-104 May 1999 Centre for Discrete Mathematics and Theoretical

### which possibility holds in an ensemble with a given a priori probability p k log 2

ALGORITHMIC INFORMATION THEORY Encyclopedia of Statistical Sciences, Volume 1, Wiley, New York, 1982, pp. 38{41 The Shannon entropy* concept of classical information theory* [9] is an ensemble notion it

### Computational Complexity: A Modern Approach. Draft of a book: Dated January 2007 Comments welcome!

i Computational Complexity: A Modern Approach Draft of a book: Dated January 2007 Comments welcome! Sanjeev Arora and Boaz Barak Princeton University complexitybook@gmail.com Not to be reproduced or distributed

### CONSTRUCTION OF THE REAL NUMBERS.

CONSTRUCTION OF THE REAL NUMBERS. IAN KIMING 1. Motivation. It will not come as a big surprise to anyone when I say that we need the real numbers in mathematics. More to the point, we need to be able to

### Kolmogorov complexity ; induction, prediction and compression

Kolmogorov complexity ; induction, prediction and compression Contents 1 Motivation for Kolmogorov complexity 1 2 Formal Definition 2 3 Trying to compute Kolmogorov complexity 3 4 Standard upper bounds

### Decidability and Undecidability

Decidability and Undecidability Major Ideas from Last Time Every TM can be converted into a string representation of itself. The encoding of M is denoted M. The universal Turing machine U TM accepts an

### Theory of Computation

Theory of Computation Dr. Sarmad Abbasi Dr. Sarmad Abbasi () Theory of Computation 1 / 33 Lecture 20: Overview Incompressible strings Minimal Length Descriptions Descriptive Complexity Dr. Sarmad Abbasi

### 1 Acceptance, Rejection, and I/O for Turing Machines

1 Acceptance, Rejection, and I/O for Turing Machines Definition 1.1 (Initial Configuration) If M = (K,Σ,δ,s,H) is a Turing machine and w (Σ {, }) then the initial configuration of M on input w is (s, w).

### Turing machines and computable functions

Turing machines and computable functions In this chapter we address the question: how can we define the class of discrete functions that are computable? Probably, most of you are familiar with a few programming

### CHAPTER 11. Introduction to Intuitionistic Logic

CHAPTER 11 Introduction to Intuitionistic Logic Intuitionistic logic has developed as a result of certain philosophical views on the foundation of mathematics, known as intuitionism. Intuitionism was originated

### CpSc 421 Homework 9 Solution

CpSc 421 Homework 9 Solution Attempt any three of the six problems below. The homework is graded on a scale of 100 points, even though you can attempt fewer or more points than that. Your recorded grade

### Creative Objectivism, a powerful alternative to Constructivism

Creative Objectivism, a powerful alternative to Constructivism Copyright c 2002 Paul P. Budnik Jr. Mountain Math Software All rights reserved Abstract It is problematic to allow reasoning about infinite

### Kolmogorov Complexity and Diophantine Approximation

Kolmogorov Complexity and Diophantine Approximation Jan Reimann Institut für Informatik Universität Heidelberg Kolmogorov Complexity and Diophantine Approximation p. 1/24 Algorithmic Information Theory

### arxiv:math/ v1 [math.nt] 17 Feb 2003

arxiv:math/0302183v1 [math.nt] 17 Feb 2003 REPRESENTATIONS OF Ω IN NUMBER THEORY: FINITUDE VERSUS PARITY TOBY ORD AND TIEN D. KIEU Abstract. We present a new method for expressing Chaitin s random real,

### Computational Models Lecture 9, Spring 2009

Slides modified by Benny Chor, based on original slides by Maurice Herlihy, Brown University. p. 1 Computational Models Lecture 9, Spring 2009 Reducibility among languages Mapping reductions More undecidable

### arxiv: v5 [cs.it] 5 Oct 2011

Numerical Evaluation of Algorithmic Complexity for Short Strings: A Glance Into the Innermost Structure of Randomness arxiv:1101.4795v5 [cs.it] 5 Oct 2011 Jean-Paul Delahaye 1, Hector Zenil 1 Email addresses:

### arxiv:cs/ v1 [cs.cc] 18 Oct 2001

arxiv:cs/0110040v1 [cs.cc] 18 Oct 2001 A New Approach to Formal Language Theory by Kolmogorov Complexity Ming Li University of Waterloo Paul M.B. Vitányi CWI and Universiteit van Amsterdam July 23, 2013

### 1 Computational Problems

Stanford University CS254: Computational Complexity Handout 2 Luca Trevisan March 31, 2010 Last revised 4/29/2010 In this lecture we define NP, we state the P versus NP problem, we prove that its formulation

### Hence, the sequence of triangular numbers is given by., the. n th square number, is the sum of the first. S n

Appendix A: The Principle of Mathematical Induction We now present an important deductive method widely used in mathematics: the principle of mathematical induction. First, we provide some historical context

### Kolmogorov Complexity in Randomness Extraction

LIPIcs Leibniz International Proceedings in Informatics Kolmogorov Complexity in Randomness Extraction John M. Hitchcock, A. Pavan 2, N. V. Vinodchandran 3 Department of Computer Science, University of

### The efficiency of identifying timed automata and the power of clocks

The efficiency of identifying timed automata and the power of clocks Sicco Verwer a,b,1,, Mathijs de Weerdt b, Cees Witteveen b a Eindhoven University of Technology, Department of Mathematics and Computer

### Kolmogorov Complexity

Kolmogorov Complexity 1 Krzysztof Zawada University of Illinois at Chicago E-mail: kzawada@uic.edu Abstract Information theory is a branch of mathematics that attempts to quantify information. To quantify

### 7.1 The Origin of Computer Science

CS125 Lecture 7 Fall 2016 7.1 The Origin of Computer Science Alan Mathison Turing (1912 1954) turing.jpg 170!201 pixels On Computable Numbers, with an Application to the Entscheidungsproblem 1936 1936:

### DRAFT. Algebraic computation models. Chapter 14

Chapter 14 Algebraic computation models Somewhat rough We think of numerical algorithms root-finding, gaussian elimination etc. as operating over R or C, even though the underlying representation of the

### Applications of Kolmogorov Complexity to Graphs

Applications of Kolmogorov Complexity to Graphs John Hearn Ran Libeskind-Hadas, Advisor Michael Orrison, Reader May, 2006 Department of Mathematics Copyright 2006 John Hearn. The author grants Harvey Mudd

### A Simple Proof of Gödel s Incompleteness Theorems

A Simple Proof of Gödel s Incompleteness Theorems Arindama Singh, Department of Mathematics, IIT Madras, Chennai-600036 Email: asingh@iitm.ac.in 1 Introduction Gödel s incompleteness theorems are considered

### Theory of Computation Lecture Notes. Problems and Algorithms. Class Information

Theory of Computation Lecture Notes Prof. Yuh-Dauh Lyuu Dept. Computer Science & Information Engineering and Department of Finance National Taiwan University Problems and Algorithms c 2004 Prof. Yuh-Dauh

### Models. Models of Computation, Turing Machines, and the Limits of Turing Computation. Effective Calculability. Motivation for Models of Computation

Turing Computation /0/ Models of Computation, Turing Machines, and the Limits of Turing Computation Bruce MacLennan Models A model is a tool intended to address a class of questions about some domain of

### Further discussion of Turing machines

Further discussion of Turing machines In this lecture we will discuss various aspects of decidable and Turing-recognizable languages that were not mentioned in previous lectures. In particular, we will

### Complexity and NP-completeness

Lecture 17 Complexity and NP-completeness Supplemental reading in CLRS: Chapter 34 As an engineer or computer scientist, it is important not only to be able to solve problems, but also to know which problems

### 1 Computational problems

80240233: Computational Complexity Lecture 1 ITCS, Tsinghua Univesity, Fall 2007 9 October 2007 Instructor: Andrej Bogdanov Notes by: Andrej Bogdanov The aim of computational complexity theory is to study

### Notes on Complexity Theory Last updated: October, Lecture 6

Notes on Complexity Theory Last updated: October, 2015 Lecture 6 Notes by Jonathan Katz, lightly edited by Dov Gordon 1 PSPACE and PSPACE-Completeness As in our previous study of N P, it is useful to identify

### Gödel's famous incompleteness theorems from the 1930's demonstrate the possibility of unprovable theorems in discrete and finite mathematics.

1 UNPROVABLE THEOREMS IN DISCRETE MATHEMATICS Harvey M. Friedman Department of Mathematics Ohio State University friedman@math.ohio-state.edu www.math.ohio-state.edu/~friedman/ April 26, 1999 An unprovable

### Minimum Message Length and Kolmogorov Complexity

Minimum Message Length and Kolmogorov Complexity C. S. WALLACE AND D. L. DOWE Computer Science, Monash University, Clayton Vic 3168 Australia Email: csw@cs.monash.edu.au The notion of algorithmic complexity

### Homework 8. a b b a b a b. two-way, read/write

Homework 8 309 Homework 8 1. Describe a TM that accepts the set {a n n is a power of 2}. Your description should be at the level of the descriptions in Lecture 29 of the TM that accepts {ww w Σ } and the

### Section 3.1: Direct Proof and Counterexample 1

Section 3.1: Direct Proof and Counterexample 1 In this chapter, we introduce the notion of proof in mathematics. A mathematical proof is valid logical argument in mathematics which shows that a given conclusion

### ON COMPUTAMBLE NUMBERS, WITH AN APPLICATION TO THE ENTSCHENIDUGSPROBLEM. Turing 1936

ON COMPUTAMBLE NUMBERS, WITH AN APPLICATION TO THE ENTSCHENIDUGSPROBLEM Turing 1936 Where are We? Ignoramus et ignorabimus Wir mussen wissen Wir werden wissen We do not know We shall not know We must know

### Primitive recursive functions: decidability problems

Primitive recursive functions: decidability problems Armando B. Matos October 24, 2014 Abstract Although every primitive recursive (PR) function is total, many problems related to PR functions are undecidable.

### Math Real Analysis

1 / 28 Math 370 - Real Analysis G.Pugh Sep 3 2013 Real Analysis 2 / 28 3 / 28 What is Real Analysis? Wikipedia: Real analysis... has its beginnings in the rigorous formulation of calculus. It is a branch

### Applications of Kolmogorov Complexity to Graphs

Applications of Kolmogorov Complexity to Graphs John Hearn Ran Libeskind-Hadas, Advisor Michael Orrison, Reader December 12, 2005 Department of Mathematics Abstract Kolmogorov complexity is a theory based

### Partial Collapses of the Σ 1 Complexity Hierarchy in Models for Fragments of Bounded Arithmetic

Partial Collapses of the Σ 1 Complexity Hierarchy in Models for Fragments of Bounded Arithmetic Zofia Adamowicz Institute of Mathematics, Polish Academy of Sciences Śniadeckich 8, 00-950 Warszawa, Poland

### Logic. Propositional Logic: Syntax

Logic Propositional Logic: Syntax Logic is a tool for formalizing reasoning. There are lots of different logics: probabilistic logic: for reasoning about probability temporal logic: for reasoning about

### Verifying whether One-Tape Turing Machines Run in Linear Time

Electronic Colloquium on Computational Complexity, Report No. 36 (2015) Verifying whether One-Tape Turing Machines Run in Linear Time David Gajser IMFM, Jadranska 19, 1000 Ljubljana, Slovenija david.gajser@fmf.uni-lj.si

### Introduction to Turing Machines. Reading: Chapters 8 & 9

Introduction to Turing Machines Reading: Chapters 8 & 9 1 Turing Machines (TM) Generalize the class of CFLs: Recursively Enumerable Languages Recursive Languages Context-Free Languages Regular Languages

### Poincaré, Heisenberg, Gödel. Some limits of scientific knowledge. Fernando Sols Universidad Complutense de Madrid

Poincaré, Heisenberg, Gödel. Some limits of scientific knowledge. Fernando Sols Universidad Complutense de Madrid Henry Poincaré (1854-1912) nonlinear dynamics Werner Heisenberg (1901-1976) uncertainty

### Chapter 2 Algorithms and Computation

Chapter 2 Algorithms and Computation In this chapter, we first discuss the principles of algorithm and computation in general framework, common both in classical and quantum computers, then we go to the

### Lecture 2. 1 More N P-Compete Languages. Notes on Complexity Theory: Fall 2005 Last updated: September, Jonathan Katz

Notes on Complexity Theory: Fall 2005 Last updated: September, 2005 Jonathan Katz Lecture 2 1 More N P-Compete Languages It will be nice to find more natural N P-complete languages. To that end, we ine

### Kvantifikator för en Dag

Philosophical Communications, Web Series, No. 35, pp. 155-162 Dept. of Philosophy, Göteborg University, Sweden ISSN 1652-0459 Kvantifikator för en Dag Essays dedicated to Dag Westerståhl on his sixtieth

### The constructible universe

The constructible universe In this set of notes I want to sketch Gödel s proof that CH is consistent with the other axioms of set theory. Gödel s argument goes well beyond this result; his identification

### Turing Machines Part III

Turing Machines Part III Announcements Problem Set 6 due now. Problem Set 7 out, due Monday, March 4. Play around with Turing machines, their powers, and their limits. Some problems require Wednesday's

### 198:538 Lecture Given on February 23rd, 1998

198:538 Lecture Given on February 23rd, 1998 Lecturer: Professor Eric Allender Scribe: Sunny Daniels November 25, 1998 1 A Result From theorem 5 of the lecture on 16th February 1998, together with the

### 6.045: Automata, Computability, and Complexity Or, Great Ideas in Theoretical Computer Science Spring, Class 10 Nancy Lynch

6.045: Automata, Computability, and Complexity Or, Great Ideas in Theoretical Computer Science Spring, 2010 Class 10 Nancy Lynch Today Final topic in computability theory: Self-Reference and the Recursion

### The Polynomial Hierarchy

The Polynomial Hierarchy Slides based on S.Aurora, B.Barak. Complexity Theory: A Modern Approach. Ahto Buldas Ahto.Buldas@ut.ee Motivation..synthesizing circuits is exceedingly difficulty. It is even

### The Limit of Humanly Knowable Mathematical Truth

The Limit of Humanly Knowable Mathematical Truth Gödel s Incompleteness Theorems, and Artificial Intelligence Santa Rosa Junior College December 12, 2015 Another title for this talk could be... An Argument

### Logical Depth and Physical Complexity

Logical Depth and Physical Complexity Charles H. Bennett IBM Research, Yorktown Heights NY 10598, USA pp. 227-257 in The Universal Turing Machine a Half-Century Survey, edited by Rolf Herken, Oxford University

### Symposium on Theoretical Aspects of Computer Science 2008 (Bordeaux), pp

Symposium on Theoretical Aspects of Computer Science 2008 (Bordeaux), pp. 373-384 www.stacs-conf.org COMPLEXITY OF SOLUTIONS OF EQUATIONS OVER SETS OF NATURAL NUMBERS ARTUR JEŻ 1 AND ALEXANDER OKHOTIN

### k-protected VERTICES IN BINARY SEARCH TREES

k-protected VERTICES IN BINARY SEARCH TREES MIKLÓS BÓNA Abstract. We show that for every k, the probability that a randomly selected vertex of a random binary search tree on n nodes is at distance k from

### 6.841/18.405J: Advanced Complexity Wednesday, February 12, Lecture Lecture 3

6.841/18.405J: Advanced Complexity Wednesday, February 12, 2003 Lecture Lecture 3 Instructor: Madhu Sudan Scribe: Bobby Kleinberg 1 The language MinDNF At the end of the last lecture, we introduced the

### Nondeterministic Finite Automata

Nondeterministic Finite Automata Mahesh Viswanathan Introducing Nondeterminism Consider the machine shown in Figure. Like a DFA it has finitely many states and transitions labeled by symbols from an input

### Propositional Logics and their Algebraic Equivalents

Propositional Logics and their Algebraic Equivalents Kyle Brooks April 18, 2012 Contents 1 Introduction 1 2 Formal Logic Systems 1 2.1 Consequence Relations......................... 2 3 Propositional Logic

### Computer Sciences Department

Computer Sciences Department 1 Reference Book: INTRODUCTION TO THE THEORY OF COMPUTATION, SECOND EDITION, by: MICHAEL SIPSER Computer Sciences Department 3 ADVANCED TOPICS IN C O M P U T A B I L I T Y

### Time, Space, and Energy in Reversible Computing. Paul Vitanyi, CWI, University of Amsterdam, National ICT Australia

Time, Space, and Energy in Reversible Computing Paul Vitanyi, CWI, University of Amsterdam, National ICT Australia Thermodynamics of Computing Heat Dissipation Input Compute Output Physical Law: 1 kt ln

### An Introduction to Gödel s Theorems

An Introduction to Gödel s Theorems In 1931, the young Kurt Gödel published his First Incompleteness Theorem, which tells us that, for any sufficiently rich theory of arithmetic, there are some arithmetical

### cse303 ELEMENTS OF THE THEORY OF COMPUTATION Professor Anita Wasilewska

cse303 ELEMENTS OF THE THEORY OF COMPUTATION Professor Anita Wasilewska LECTURE 13 CHAPTER 4 TURING MACHINES 1. The definition of Turing machine 2. Computing with Turing machines 3. Extensions of Turing

### On the non-existence of maximal inference degrees for language identification

On the non-existence of maximal inference degrees for language identification Sanjay Jain Institute of Systems Science National University of Singapore Singapore 0511 Republic of Singapore sanjay@iss.nus.sg

### Lecture 2: Syntax. January 24, 2018

Lecture 2: Syntax January 24, 2018 We now review the basic definitions of first-order logic in more detail. Recall that a language consists of a collection of symbols {P i }, each of which has some specified

### Kolmogorov complexity: a primer. Alexander Shen, LIF CNRS & Univ. Aix Marseille. June 26, 2009

Kolmogorov complexity: a primer Alexander Shen, LIF CNRS & Univ. Aix Marseille June 26, 2009 Information density The same English text uses four times more space in UTF-32 compared to ASCII Information

### . As the binomial coefficients are integers we have that. 2 n(n 1).

Math 580 Homework. 1. Divisibility. Definition 1. Let a, b be integers with a 0. Then b divides b iff there is an integer k such that b = ka. In the case we write a b. In this case we also say a is a factor

### INFINITY: CARDINAL NUMBERS

INFINITY: CARDINAL NUMBERS BJORN POONEN 1 Some terminology of set theory N := {0, 1, 2, 3, } Z := {, 2, 1, 0, 1, 2, } Q := the set of rational numbers R := the set of real numbers C := the set of complex

### Parallel Learning of Automatic Classes of Languages

Parallel Learning of Automatic Classes of Languages Sanjay Jain a,1, Efim Kinber b a School of Computing, National University of Singapore, Singapore 117417. b Department of Computer Science, Sacred Heart

### Average Case Analysis of QuickSort and Insertion Tree Height using Incompressibility

Average Case Analysis of QuickSort and Insertion Tree Height using Incompressibility Tao Jiang, Ming Li, Brendan Lucier September 26, 2005 Abstract In this paper we study the Kolmogorov Complexity of a

### Universal Prediction of Selected Bits

Universal Prediction of Selected Bits Tor Lattimore and Marcus Hutter and Vaibhav Gavane Australian National University tor.lattimore@anu.edu.au Australian National University and ETH Zürich marcus.hutter@anu.edu.au

### Turing Machines. The Language Hierarchy. Context-Free Languages. Regular Languages. Courtesy Costas Busch - RPI 1

Turing Machines a n b n c The anguage Hierarchy n? ww? Context-Free anguages a n b n egular anguages a * a *b* ww Courtesy Costas Busch - PI a n b n c n Turing Machines anguages accepted by Turing Machines

### Discrete Mathematics

Discrete Mathematics Chih-Wei Yi Dept. of Computer Science National Chiao Tung University March 9, 2009 Overview of ( 1.5-1.7, ~2 hours) Methods of mathematical argument (i.e., proof methods) can be formalized

### Enhancing Active Automata Learning by a User Log Based Metric

Master Thesis Computing Science Radboud University Enhancing Active Automata Learning by a User Log Based Metric Author Petra van den Bos First Supervisor prof. dr. Frits W. Vaandrager Second Supervisor