# Kolmogorov complexity and its applications

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Spring, 2009 Kolmogorov complexity and its applications Paul Vitanyi Computer Science University of Amsterdam

2 We live in an information society. Information science is our profession. But do you know what is information, mathematically, and how to use it to prove theorems? You will, by the end of the term.

3 Examples Average case analysis of Shellsort. Open since What is the distance between two pieces of information carrying entities? For example, distance from an internet query to an answer.

4 Lecture 1. History and Definition History Intuition and ideas in the past Inventors Basic mathematical theory The course. Textbook: Li-Vitanyi: An introduction to Kolmogorov complexity and its applications, preferably the third edition! Homework every week about last lecture, mid-term and final exam (or possibly individual project and presentation).

5 1. Intuition & history What is the information content of an individual string? (n 1 s) π = n = Champernowne s number: is normal in scale 10 (every block has same frequency) All these numbers share one commonality: there are small programs to generate them. Shannon s information theory does not help here.

6 1903: An interesting year This and the next two pages were stolen from Lance Fortnow

7 1903: An interesting year Kolmogorov Church von Neumann

8 Andrey Nikolaevich Kolmogorov (1903, Tambov, Russia 1987 Moscow) Measure Theory Probability Analysis Intuitionistic Logic Cohomology Dynamical Systems Hydrodynamics Kolmogorov complexity

9

10

11 When there were no digital cameras (1987).

12 A story of Dr. Samuel Johnson Dr. Beattie observed, as something remarkable which had happened to him, that he chanced to see both No.1 and No.1000 hackney-coaches. Why sir, said Johnson there is an equal chance for one s seeing those two numbers as any other two. Boswell s Life of Johnson

13 The case of cheating casino Bob proposes to flip a coin with Alice: Alice wins a dollar if Heads; Bob wins a dollar if Tails Result: TTTTTT. 100 Tails in a roll. Alice lost \$100. She feels being cheated.

14 Alice goes to the court Alice complains: T 100 is not random. Bob asks Alice to produce a random coin flip sequence. Alice flipped her coin and got THTTHHTHTHHHTTTTH But Bob claims Alice s sequence has probability 2-100, and so does his. How do we define randomness?

15 2. Roots of Kolmogorov complexity and preliminaries (1) Foundations of Probability P. Laplace: a sequence is extraordinary (nonrandom) because it contains regularity (which is rare) von Mises notion of a random sequence S: lim n { #(1) in n-prefix of S}/n =p, 0<p<1 The above holds for any subsequence of S selected by an admissible selection rule. If `admissible rule is any partial function then there are no random sequences. A. Wald: countably many admissible selection rules. Then there are random sequences. A. Church: recursive selection functions J. Ville: von Mises-Wald-Church random sequence does not satisfy all laws of randomness.

16 Roots (2) Information Theory. Shannon theory is on an ensemble. But what is information in an individual object? (3) Inductive inference. Bayesian approach using universal prior distribution (4) Shannon s State x Symbol (Turing machine) complexity.

17 Preliminaries and Notations Strings: x, y, z. Usually binary. x=x 1 x 2... an infinite binary sequence x i:j =x i x i+1 x j x is number of bits in x. Textbook uses l(x). Sets, A, B, C A, number of elements in set A. Textbook uses d(a). K-complexity vs C-complexity, names etc. I assume you know Turing machines, universal TM s, basic facts...

18 3. Mathematical Theory Solomonoff (1960)-Kolmogorov (1965)-Chaitin (1969): The amount of information in a string is the size of the smallest program of an optimal Universal TM U generating that string. C (x) = min { p : U(p) = x } U p Invariance Theorem: It does not matter which optimal universal Turing machine U we choose. I.e. all universal encoding methods are ok.

19 Proof of the Invariance theorem Fix an effective enumeration of all Turing machines (TM s): T 1, T 2, Define C = min { p : T(p) = x} T p U is an optimal universal TM such that (p produces x) U(1 n 0p) = T n (p) Then for all x: C U (x) C Tn (x) + n+1, and C U (x) C U (x) c. Fixing U, we write C(x) instead of C U (x). QED Formal statement of the Invariance Theorem: There exists a computable function S 0 such that for all computable functions S, there is a constant c S such that for all strings x ε {0,1} * C S0 (x) C S (x) + c S

20 It has many applications Mathematics --- probability theory, logic, statistics. Physics --- chaos, thermodynamics. Computer Science average case analysis, inductive inference and learning, shared information between documents, data mining and clustering, incompressibility method -- examples: Prime number theorem Goedel s incompleteness Shellsort average case Heapsort average case Circuit complexity Lower bounds on combinatorics, graphs,turing machine computations, formal languages, communication complexity, routing Philosophy, biology, cognition, etc randomness, inference, learning, complex systems, sequence similarity Information theory information in individual objects, information distance Classifying objects: documents, genomes Query Answering systems

21 Mathematical Theory cont. Intuitively: C(x)= length of shortest description of x Define conditional Kolmogorov complexity similarly, with C(x y)=length of shortest description of x given y. Examples C(xx) = C(x) + O(1) C(xy) C(x) + C(y) + O(log(min{C(x),C(y)}) C(1 n ) O(logn) C(π 1:n ) O(logn); C(π 1:n n) O(1) For all x, C(x) x +O(1) C(x x) = O(1) C(x ε) = C(x); C(ε x)=o(1)

22 3.1 Basics Incompressibility: For constant c>0, a string x ε {0,1} * is c-incompressible if C(x) x -c. For constant c, we often simply say that x is incompressible. (We will call incompressible strings random strings.) Lemma. There are at least 2 n 2 n-c +1 c-incompressible strings of length n. Proof. There are only k=0,,n-c-1 2 k = 2 n-c -1 programs with length less than n-c. Hence only that many strings (out of total 2 n strings of length n) can have shorter programs (descriptions) than n-c. QED.

23 Facts If x=uvw is incompressible, then C(v) v - O(log x ). Proof. C(uvw) = uvw uw +C(v)+ O(log u ) +O(log C(v)). If p is the shortest program for x, then C(p) p - O(1) C(x p) = O(1) but C(p x) C( p )+O(1) (optimal because of the Halting Problem!) If a subset A of {0,1}* is recursively enumerable (r.e.) (the elements of A can be listed by a Turing machine), and A is sparse ( A =n p(n) for some polynomial p), then for all x in A, x =n, C(x) O(log p(n) ) + O(C(n)) +O( A )= O(log n).

24 3.2 Asymptotics Enumeration of binary strings: 0,1,00,01,10, mapping to natural numbers 0, 1, 2, 3, C(x) as x Define m(x) to be the monotonic lower bound of C(x) curve (as natural number x ). Then m(x), as x, and m(x) < Q(x) for all unbounded computable Q. Nonmonotonicity: for x=yz, it does not imply that C(y) C(x)+O(1).

25 Graph of C(x) for integer x. Function m(x) is greatest monotonic non-decreasing lower bound.

26 Length-conditional complexity. Let x=x_1...x_n. Self-delimiting codes are x =1^n 0x with x =2n+1, and x = 1^{ n }0nx with x = n+2 n +1 ( n =log n). n-strings are x s of the form x=n with n= x. Note that n =2 log n +1. So, for every n, C(x n)=o(1) for all n-strings.

27 Graph of C(x l(x)). Function m(x) is greatest monotonic non-decreasing lower bound.

28 3.3 Properties Theorem (Kolmogorov) (i) C(x) is not partially recursive. That is, there is no Turing machine M s.t. M accepts (x,k) if C(x) k and undefined otherwise. (ii) However, there is H(t,x) such that H(t+1,x) H(t,x) and lim t H(t,x)=C(x) where H(t,x) is total recursive. Proof. (i) If such M exists, then design M as follows. M simulates M on input (x,n), for all x =n in parallel (one step each), and outputs the first x such that M says `yes. Choose n >> M. Thus we have a contradiction: C(x) n by M, but M outputs x hence x =n >> M C(x) n. (ii) TM with program for x running for t steps defines H(t,x). QED

29 3.4 Godel s Theorem Theorem. The statement x is random (=incompressible) is undecidable for all but finitely many x. Proof (J. Barzdins, G. Chaitin). Let F be an axiomatic theory (sound, consistent, containing PA). C(F)= C. If the theorem is false and statement x is random is provable in F, then we can enumerate all proofs in F to find a proof of x is random. Consider x s with (1) x >> C+O(log x ), and output (first) random such x. Then (2) C(x) < C +O(log x ) But the proof for x is random implies that (3) C(x) x. Now (1)+(2)+(3) yields a contradiction, C+O(log x ) >> C+O(log x ). QED

30 3.5 Barzdin s Lemma A characteristic sequence of set A is an infinite binary sequence χ=χ 1 χ 2, χ i =1 iff iεa. Theorem. (i) The characteristic sequence χ of an r.e. set A satisfies C(χ 1:n n) log n+c A for all n. (ii) There is an r.e. set such that C(χ 1:n ) log n for all n. Proof. (i) Use the number m of 1 s in the prefix χ 1:n as termination condition [C(m) log n+o(1)]. (ii) By diagonalization. Let U be the universal TM. Define χ=χ 1 χ 2, by χ i =1 if the i-th bit output by U(i)< equals 0, otherwise χ i =0. χ defines an r.e. set. Suppose, for some n, we have C(χ 1:n )<log n. Then, there is a program p such that for all i n we have U(p,i)= χ i and p < log n, hence p<n. But U(p,p) not equal χ p by definition. QED

### Kolmogorov complexity and its applications

CS860, Winter, 2010 Kolmogorov complexity and its applications Ming Li School of Computer Science University of Waterloo http://www.cs.uwaterloo.ca/~mli/cs860.html We live in an information society. Information

### CISC 876: Kolmogorov Complexity

March 27, 2007 Outline 1 Introduction 2 Definition Incompressibility and Randomness 3 Prefix Complexity Resource-Bounded K-Complexity 4 Incompressibility Method Gödel s Incompleteness Theorem 5 Outline

### Sophistication Revisited

Sophistication Revisited Luís Antunes Lance Fortnow August 30, 007 Abstract Kolmogorov complexity measures the ammount of information in a string as the size of the shortest program that computes the string.

### Lecture 13: Foundations of Math and Kolmogorov Complexity

6.045 Lecture 13: Foundations of Math and Kolmogorov Complexity 1 Self-Reference and the Recursion Theorem 2 Lemma: There is a computable function q : Σ* Σ* such that for every string w, q(w) is the description

### Computability Theory

Computability Theory Cristian S. Calude May 2012 Computability Theory 1 / 1 Bibliography M. Sipser. Introduction to the Theory of Computation, PWS 1997. (textbook) Computability Theory 2 / 1 Supplementary

### Time, Space, and Energy in Reversible Computing. Paul Vitanyi, CWI, University of Amsterdam, National ICT Australia

Time, Space, and Energy in Reversible Computing Paul Vitanyi, CWI, University of Amsterdam, National ICT Australia Thermodynamics of Computing Heat Dissipation Input Compute Output Physical Law: 1 kt ln

### 1 Computational Problems

Stanford University CS254: Computational Complexity Handout 2 Luca Trevisan March 31, 2010 Last revised 4/29/2010 In this lecture we define NP, we state the P versus NP problem, we prove that its formulation

### Introduction to Kolmogorov Complexity

Introduction to Kolmogorov Complexity Marcus Hutter Canberra, ACT, 0200, Australia http://www.hutter1.net/ ANU Marcus Hutter - 2 - Introduction to Kolmogorov Complexity Abstract In this talk I will give

### 17.1 The Halting Problem

CS125 Lecture 17 Fall 2016 17.1 The Halting Problem Consider the HALTING PROBLEM (HALT TM ): Given a TM M and w, does M halt on input w? Theorem 17.1 HALT TM is undecidable. Suppose HALT TM = { M,w : M

### CS154, Lecture 12: Kolmogorov Complexity: A Universal Theory of Data Compression

CS154, Lecture 12: Kolmogorov Complexity: A Universal Theory of Data Compression Rosencrantz & Guildenstern Are Dead (Tom Stoppard) Rigged Lottery? And the winning numbers are: 1, 2, 3, 4, 5, 6 But is

### FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY

15-453 FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY KOLMOGOROV-CHAITIN (descriptive) COMPLEXITY TUESDAY, MAR 18 CAN WE QUANTIFY HOW MUCH INFORMATION IS IN A STRING? A = 01010101010101010101010101010101

### The axiomatic power of Kolmogorov complexity

The axiomatic power of Kolmogorov complexity Laurent Bienvenu 1, Andrei Romashchenko 2, Alexander Shen 2, Antoine Taveneaux 1, and Stijn Vermeeren 3 1 LIAFA, CNRS & Université Paris 7 2 LIRMM, CNRS & Université

### Finish K-Complexity, Start Time Complexity

6.045 Finish K-Complexity, Start Time Complexity 1 Kolmogorov Complexity Definition: The shortest description of x, denoted as d(x), is the lexicographically shortest string such that M(w) halts

### Computability Theory. CS215, Lecture 6,

Computability Theory CS215, Lecture 6, 2000 1 The Birth of Turing Machines At the end of the 19th century, Gottlob Frege conjectured that mathematics could be built from fundamental logic In 1900 David

### Introduction to Turing Machines. Reading: Chapters 8 & 9

Introduction to Turing Machines Reading: Chapters 8 & 9 1 Turing Machines (TM) Generalize the class of CFLs: Recursively Enumerable Languages Recursive Languages Context-Free Languages Regular Languages

### Decision Problems with TM s. Lecture 31: Halting Problem. Universe of discourse. Semi-decidable. Look at following sets: CSCI 81 Spring, 2012

Decision Problems with TM s Look at following sets: Lecture 31: Halting Problem CSCI 81 Spring, 2012 Kim Bruce A TM = { M,w M is a TM and w L(M)} H TM = { M,w M is a TM which halts on input w} TOTAL TM

### Chapter 2 Algorithms and Computation

Chapter 2 Algorithms and Computation In this chapter, we first discuss the principles of algorithm and computation in general framework, common both in classical and quantum computers, then we go to the

### COS597D: Information Theory in Computer Science October 19, Lecture 10

COS597D: Information Theory in Computer Science October 9, 20 Lecture 0 Lecturer: Mark Braverman Scribe: Andrej Risteski Kolmogorov Complexity In the previous lectures, we became acquainted with the concept

### Algorithmic Probability

Algorithmic Probability From Scholarpedia From Scholarpedia, the free peer-reviewed encyclopedia p.19046 Curator: Marcus Hutter, Australian National University Curator: Shane Legg, Dalle Molle Institute

### Theory of Computation

Theory of Computation Dr. Sarmad Abbasi Dr. Sarmad Abbasi () Theory of Computation 1 / 33 Lecture 20: Overview Incompressible strings Minimal Length Descriptions Descriptive Complexity Dr. Sarmad Abbasi

### Lecture 11: Quantum Information III - Source Coding

CSCI5370 Quantum Computing November 25, 203 Lecture : Quantum Information III - Source Coding Lecturer: Shengyu Zhang Scribe: Hing Yin Tsang. Holevo s bound Suppose Alice has an information source X that

### Algorithmic Information Theory

Algorithmic Information Theory Peter D. Grünwald CWI, P.O. Box 94079 NL-1090 GB Amsterdam, The Netherlands E-mail: pdg@cwi.nl Paul M.B. Vitányi CWI, P.O. Box 94079 NL-1090 GB Amsterdam The Netherlands

### Lecture Notes: The Halting Problem; Reductions

Lecture Notes: The Halting Problem; Reductions COMS W3261 Columbia University 20 Mar 2012 1 Review Key point. Turing machines can be encoded as strings, and other Turing machines can read those strings

### Computer Sciences Department

Computer Sciences Department 1 Reference Book: INTRODUCTION TO THE THEORY OF COMPUTATION, SECOND EDITION, by: MICHAEL SIPSER Computer Sciences Department 3 ADVANCED TOPICS IN C O M P U T A B I L I T Y

### CS 125 Section #10 (Un)decidability and Probability November 1, 2016

CS 125 Section #10 (Un)decidability and Probability November 1, 2016 1 Countability Recall that a set S is countable (either finite or countably infinite) if and only if there exists a surjective mapping

### Logic: The Big Picture

Logic: The Big Picture A typical logic is described in terms of syntax: what are the legitimate formulas semantics: under what circumstances is a formula true proof theory/ axiomatization: rules for proving

### CS154, Lecture 10: Rice s Theorem, Oracle Machines

CS154, Lecture 10: Rice s Theorem, Oracle Machines Moral: Analyzing Programs is Really, Really Hard But can we more easily tell when some program analysis problem is undecidable? Problem 1 Undecidable

### Kolmogorov Complexity and Diophantine Approximation

Kolmogorov Complexity and Diophantine Approximation Jan Reimann Institut für Informatik Universität Heidelberg Kolmogorov Complexity and Diophantine Approximation p. 1/24 Algorithmic Information Theory

### Average Case Analysis of QuickSort and Insertion Tree Height using Incompressibility

Average Case Analysis of QuickSort and Insertion Tree Height using Incompressibility Tao Jiang, Ming Li, Brendan Lucier September 26, 2005 Abstract In this paper we study the Kolmogorov Complexity of a

### 16.1 Countability. CS125 Lecture 16 Fall 2014

CS125 Lecture 16 Fall 2014 16.1 Countability Proving the non-existence of algorithms for computational problems can be very difficult. Indeed, we do not know how to prove P NP. So a natural question is

### A version of for which ZFC can not predict a single bit Robert M. Solovay May 16, Introduction In [2], Chaitin introd

CDMTCS Research Report Series A Version of for which ZFC can not Predict a Single Bit Robert M. Solovay University of California at Berkeley CDMTCS-104 May 1999 Centre for Discrete Mathematics and Theoretical

### Recovery Based on Kolmogorov Complexity in Underdetermined Systems of Linear Equations

Recovery Based on Kolmogorov Complexity in Underdetermined Systems of Linear Equations David Donoho Department of Statistics Stanford University Email: donoho@stanfordedu Hossein Kakavand, James Mammen

### Derandomizing from Random Strings

Derandomizing from Random Strings Harry Buhrman CWI and University of Amsterdam buhrman@cwi.nl Lance Fortnow Northwestern University fortnow@northwestern.edu Michal Koucký Institute of Mathematics, AS

### FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY

15-453 FORMAL LANGUAGES, AUTOMATA AND COMPUTABILITY THURSDAY APRIL 3 REVIEW for Midterm TUESDAY April 8 Definition: A Turing Machine is a 7-tuple T = (Q, Σ, Γ, δ, q, q accept, q reject ), where: Q is a

### The Turing Machine. Computability. The Church-Turing Thesis (1936) Theory Hall of Fame. Theory Hall of Fame. Undecidability

The Turing Machine Computability Motivating idea Build a theoretical a human computer Likened to a human with a paper and pencil that can solve problems in an algorithmic way The theoretical provides a

### MTAT Complexity Theory October 13th-14th, Lecture 6

MTAT.07.004 Complexity Theory October 13th-14th, 2011 Lecturer: Peeter Laud Lecture 6 Scribe(s): Riivo Talviste 1 Logarithmic memory Turing machines working in logarithmic space become interesting when

### Lecture 20: conp and Friends, Oracles in Complexity Theory

6.045 Lecture 20: conp and Friends, Oracles in Complexity Theory 1 Definition: conp = { L L NP } What does a conp computation look like? In NP algorithms, we can use a guess instruction in pseudocode:

### Final Exam Comments. UVa - cs302: Theory of Computation Spring < Total

UVa - cs302: Theory of Computation Spring 2008 Final Exam Comments < 50 50 59 60 69 70 79 80 89 90 94 95-102 Total 2 6 8 22 16 16 12 Problem 1: Short Answers. (20) For each question, provide a correct,

### arxiv:cs/ v1 [cs.cc] 18 Oct 2001

arxiv:cs/0110040v1 [cs.cc] 18 Oct 2001 A New Approach to Formal Language Theory by Kolmogorov Complexity Ming Li University of Waterloo Paul M.B. Vitányi CWI and Universiteit van Amsterdam July 23, 2013

### ON COMPUTAMBLE NUMBERS, WITH AN APPLICATION TO THE ENTSCHENIDUGSPROBLEM. Turing 1936

ON COMPUTAMBLE NUMBERS, WITH AN APPLICATION TO THE ENTSCHENIDUGSPROBLEM Turing 1936 Where are We? Ignoramus et ignorabimus Wir mussen wissen Wir werden wissen We do not know We shall not know We must know

### 1 Acceptance, Rejection, and I/O for Turing Machines

1 Acceptance, Rejection, and I/O for Turing Machines Definition 1.1 (Initial Configuration) If M = (K,Σ,δ,s,H) is a Turing machine and w (Σ {, }) then the initial configuration of M on input w is (s, w).

### which possibility holds in an ensemble with a given a priori probability p k log 2

ALGORITHMIC INFORMATION THEORY Encyclopedia of Statistical Sciences, Volume 1, Wiley, New York, 1982, pp. 38{41 The Shannon entropy* concept of classical information theory* [9] is an ensemble notion it

### 1 Randomized Computation

CS 6743 Lecture 17 1 Fall 2007 1 Randomized Computation Why is randomness useful? Imagine you have a stack of bank notes, with very few counterfeit ones. You want to choose a genuine bank note to pay at

### 3 Self-Delimiting Kolmogorov complexity

3 Self-Delimiting Kolmogorov complexity 3. Prefix codes A set is called a prefix-free set (or a prefix set) if no string in the set is the proper prefix of another string in it. A prefix set cannot therefore

### A statistical mechanical interpretation of algorithmic information theory

A statistical mechanical interpretation of algorithmic information theory Kohtaro Tadaki Research and Development Initiative, Chuo University 1 13 27 Kasuga, Bunkyo-ku, Tokyo 112-8551, Japan. E-mail: tadaki@kc.chuo-u.ac.jp

### Kolmogorov Complexity in Randomness Extraction

LIPIcs Leibniz International Proceedings in Informatics Kolmogorov Complexity in Randomness Extraction John M. Hitchcock, A. Pavan 2, N. V. Vinodchandran 3 Department of Computer Science, University of

### About the relationship between formal logic and complexity classes

About the relationship between formal logic and complexity classes Working paper Comments welcome; my email: armandobcm@yahoo.com Armando B. Matos October 20, 2013 1 Introduction We analyze a particular

### Peano Arithmetic. CSC 438F/2404F Notes (S. Cook) Fall, Goals Now

CSC 438F/2404F Notes (S. Cook) Fall, 2008 Peano Arithmetic Goals Now 1) We will introduce a standard set of axioms for the language L A. The theory generated by these axioms is denoted PA and called Peano

### On the coinductive nature of centralizers

On the coinductive nature of centralizers Charles Grellois INRIA & University of Bologna Séminaire du LIFO Jan 16, 2017 Charles Grellois (INRIA & Bologna) On the coinductive nature of centralizers Jan

### Lecture 16: Time Complexity and P vs NP

6.045 Lecture 16: Time Complexity and P vs NP 1 Time-Bounded Complexity Classes Definition: TIME(t(n)) = { L there is a Turing machine M with time complexity O(t(n)) so that L = L(M) } = { L L is a language

### Turing Machines Part III

Turing Machines Part III Announcements Problem Set 6 due now. Problem Set 7 out, due Monday, March 4. Play around with Turing machines, their powers, and their limits. Some problems require Wednesday's

### Algorithmic Information Theory and Kolmogorov Complexity

Algorithmic Information Theory and Kolmogorov Complexity Alexander Shen, Uppsala University, Computing Science Department, Independent University of Moscow, shen@mccme.ru November 19, 2007 Abstract This

### CSE 105 Theory of Computation

CSE 105 Theory of Computation http://www.jflap.org/jflaptmp/ Professor Jeanne Ferrante 1 Undecidability Today s Agenda Review and More Problems A Non-TR Language Reminders and announcements: HW 7 (Last!!)

### (a) Definition of TMs. First Problem of URMs

Sec. 4: Turing Machines First Problem of URMs (a) Definition of the Turing Machine. (b) URM computable functions are Turing computable. (c) Undecidability of the Turing Halting Problem That incrementing

### 2 P vs. NP and Diagonalization

2 P vs NP and Diagonalization CS 6810 Theory of Computing, Fall 2012 Instructor: David Steurer (sc2392) Date: 08/28/2012 In this lecture, we cover the following topics: 1 3SAT is NP hard; 2 Time hierarchies;

### ,

Kolmogorov Complexity Carleton College, CS 254, Fall 2013, Prof. Joshua R. Davis based on Sipser, Introduction to the Theory of Computation 1. Introduction Kolmogorov complexity is a theory of lossless

### 1 Computational problems

80240233: Computational Complexity Lecture 1 ITCS, Tsinghua Univesity, Fall 2007 9 October 2007 Instructor: Andrej Bogdanov Notes by: Andrej Bogdanov The aim of computational complexity theory is to study

### Primitive recursive functions: decidability problems

Primitive recursive functions: decidability problems Armando B. Matos October 24, 2014 Abstract Although every primitive recursive (PR) function is total, many problems related to PR functions are undecidable.

### Game interpretation of Kolmogorov complexity

Game interpretation of Kolmogorov complexity Andrej A. Muchnik, Ilya Mezhirov, Alexander Shen, Nikolay Vereshchagin Abstract arxiv:1003.4712v1 [math.lo] 24 Mar 2010 The Kolmogorov complexity function K

### Time Complexity (1) CSCI Spring Original Slides were written by Dr. Frederick W Maier. CSCI 2670 Time Complexity (1)

Time Complexity (1) CSCI 2670 Original Slides were written by Dr. Frederick W Maier Spring 2014 Time Complexity So far we ve dealt with determining whether or not a problem is decidable. But even if it

### Theory of Computation

Theory of Computation (Feodor F. Dragan) Department of Computer Science Kent State University Spring, 2018 Theory of Computation, Feodor F. Dragan, Kent State University 1 Before we go into details, what

### The PRIMES 2014 problem set

Dear PRIMES applicant! The PRIMES 24 problem set This is the PRIMES 24 problem set. Please send us your solutions as part of your PRIMES application by December, 24. For complete rules, see http://web.mit.edu/primes/apply.shtml

### Lecture 2. 1 More N P-Compete Languages. Notes on Complexity Theory: Fall 2005 Last updated: September, Jonathan Katz

Notes on Complexity Theory: Fall 2005 Last updated: September, 2005 Jonathan Katz Lecture 2 1 More N P-Compete Languages It will be nice to find more natural N P-complete languages. To that end, we ine

### Regular Languages and Finite Automata

Regular Languages and Finite Automata Theorem: Every regular language is accepted by some finite automaton. Proof: We proceed by induction on the (length of/structure of) the description of the regular

### Algorithmic probability, Part 1 of n. A presentation to the Maths Study Group at London South Bank University 09/09/2015

Algorithmic probability, Part 1 of n A presentation to the Maths Study Group at London South Bank University 09/09/2015 Motivation Effective clustering the partitioning of a collection of objects such

### Theory of Computation Lecture Notes. Problems and Algorithms. Class Information

Theory of Computation Lecture Notes Prof. Yuh-Dauh Lyuu Dept. Computer Science & Information Engineering and Department of Finance National Taiwan University Problems and Algorithms c 2004 Prof. Yuh-Dauh

### Lecture Notes 1 Basic Concepts of Mathematics MATH 352

Lecture Notes 1 Basic Concepts of Mathematics MATH 352 Ivan Avramidi New Mexico Institute of Mining and Technology Socorro, NM 87801 June 3, 2004 Author: Ivan Avramidi; File: absmath.tex; Date: June 11,

### Computational Models Lecture 9, Spring 2009

Slides modified by Benny Chor, based on original slides by Maurice Herlihy, Brown University. p. 1 Computational Models Lecture 9, Spring 2009 Reducibility among languages Mapping reductions More undecidable

### cse303 ELEMENTS OF THE THEORY OF COMPUTATION Professor Anita Wasilewska

cse303 ELEMENTS OF THE THEORY OF COMPUTATION Professor Anita Wasilewska LECTURE 1 Course Web Page www3.cs.stonybrook.edu/ cse303 The webpage contains: lectures notes slides; very detailed solutions to

### Notes on Complexity Theory Last updated: November, Lecture 10

Notes on Complexity Theory Last updated: November, 2015 Lecture 10 Notes by Jonathan Katz, lightly edited by Dov Gordon. 1 Randomized Time Complexity 1.1 How Large is BPP? We know that P ZPP = RP corp

### 9. PSPACE 9. PSPACE. PSPACE complexity class quantified satisfiability planning problem PSPACE-complete

Geography game Geography. Alice names capital city c of country she is in. Bob names a capital city c' that starts with the letter on which c ends. Alice and Bob repeat this game until one player is unable

### CSCE 551: Chin-Tser Huang. University of South Carolina

CSCE 551: Theory of Computation Chin-Tser Huang huangct@cse.sc.edu University of South Carolina Church-Turing Thesis The definition of the algorithm came in the 1936 papers of Alonzo Church h and Alan

### Lecture 17. In this lecture, we will continue our discussion on randomization.

ITCS:CCT09 : Computational Complexity Theory May 11, 2009 Lecturer: Jayalal Sarma M.N. Lecture 17 Scribe: Hao Song In this lecture, we will continue our discussion on randomization. 1 BPP and the Polynomial

### Kolmogorov Complexity

Kolmogorov Complexity Davide Basilio Bartolini University of Illinois at Chicago Politecnico di Milano dbarto3@uic.edu davide.bartolini@mail.polimi.it 1 Abstract What follows is a survey of the field of

### 9. PSPACE. PSPACE complexity class quantified satisfiability planning problem PSPACE-complete

9. PSPACE PSPACE complexity class quantified satisfiability planning problem PSPACE-complete Lecture slides by Kevin Wayne Copyright 2005 Pearson-Addison Wesley Copyright 2013 Kevin Wayne http://www.cs.princeton.edu/~wayne/kleinberg-tardos

### arxiv:math/ v1 [math.nt] 17 Feb 2003

arxiv:math/0302183v1 [math.nt] 17 Feb 2003 REPRESENTATIONS OF Ω IN NUMBER THEORY: FINITUDE VERSUS PARITY TOBY ORD AND TIEN D. KIEU Abstract. We present a new method for expressing Chaitin s random real,

### Computability and Complexity Theory

Discrete Math for Bioinformatics WS 09/10:, by A Bockmayr/K Reinert, January 27, 2010, 18:39 9001 Computability and Complexity Theory Computability and complexity Computability theory What problems can

### Bound and Free Variables. Theorems and Proofs. More valid formulas involving quantifiers:

Bound and Free Variables More valid formulas involving quantifiers: xp(x) x P(x) Replacing P by P, we get: x P(x) x P(x) Therefore x P(x) xp(x) Similarly, we have xp(x) x P(x) x P(x) xp(x) i(i 2 > i) is

### Lecture notes on descriptional complexity and randomness

Lecture notes on descriptional complexity and randomness Peter Gács Boston University gacs@bu.edu A didactical survey of the foundations of Algorithmic Information Theory. These notes are short on motivation,

### Decidability and Undecidability

Decidability and Undecidability Major Ideas from Last Time Every TM can be converted into a string representation of itself. The encoding of M is denoted M. The universal Turing machine U TM accepts an

### CSE 105 THEORY OF COMPUTATION

CSE 105 THEORY OF COMPUTATION "Winter" 2018 http://cseweb.ucsd.edu/classes/wi18/cse105-ab/ Today's learning goals Sipser Ch 4.2 Trace high-level descriptions of algorithms for computational problems. Use

### Polynomial-Time Reductions

Reductions 1 Polynomial-Time Reductions Classify Problems According to Computational Requirements Q. Which problems will we be able to solve in practice? A working definition. [von Neumann 1953, Godel

### 7.1 The Origin of Computer Science

CS125 Lecture 7 Fall 2016 7.1 The Origin of Computer Science Alan Mathison Turing (1912 1954) turing.jpg 170!201 pixels On Computable Numbers, with an Application to the Entscheidungsproblem 1936 1936:

### Stochasticity in Algorithmic Statistics for Polynomial Time

Stochasticity in Algorithmic Statistics for Polynomial Time Alexey Milovanov 1 and Nikolay Vereshchagin 2 1 National Research University Higher School of Economics, Moscow, Russia almas239@gmail.com 2

### Definition: conp = { L L NP } What does a conp computation look like?

Space Complexity 28 Definition: conp = { L L NP } What does a conp computation look like? In NP algorithms, we can use a guess instruction in pseudocode: Guess string y of x k length and the machine accepts

### 2 Natural Proofs: a barrier for proving circuit lower bounds

Topics in Theoretical Computer Science April 4, 2016 Lecturer: Ola Svensson Lecture 6 (Notes) Scribes: Ola Svensson Disclaimer: These notes were written for the lecturer only and may contain inconsistent

### 198:538 Lecture Given on February 23rd, 1998

198:538 Lecture Given on February 23rd, 1998 Lecturer: Professor Eric Allender Scribe: Sunny Daniels November 25, 1998 1 A Result From theorem 5 of the lecture on 16th February 1998, together with the

### CS 361 Meeting 26 11/10/17

CS 361 Meeting 26 11/10/17 1. Homework 8 due Announcements A Recognizable, but Undecidable Language 1. Last class, I presented a brief, somewhat inscrutable proof that the language A BT M = { M w M is

### CS 154, Lecture 4: Limitations on DFAs (I), Pumping Lemma, Minimizing DFAs

CS 154, Lecture 4: Limitations on FAs (I), Pumping Lemma, Minimizing FAs Regular or Not? Non-Regular Languages = { w w has equal number of occurrences of 01 and 10 } REGULAR! C = { w w has equal number

### Lecture 24: Randomized Complexity, Course Summary

6.045 Lecture 24: Randomized Complexity, Course Summary 1 1/4 1/16 1/4 1/4 1/32 1/16 1/32 Probabilistic TMs 1/16 A probabilistic TM M is a nondeterministic TM where: Each nondeterministic step is called

### EPISTEMOLOGY, COMPUTATION AND THE LAWS OF PHYSICS

EPISTEMOLOGY, COMPUTATION AND THE LAWS OF PHYSICS David H. Wolpert NASA Ames Research Center David.H.Wolpert@nasa.gov http://ti.arc.nasa.gov/people/dhw/ Walter Read Mathematics Department, CS Fresno NASA-ARC-03-0667

### Creative Objectivism, a powerful alternative to Constructivism

Creative Objectivism, a powerful alternative to Constructivism Copyright c 2002 Paul P. Budnik Jr. Mountain Math Software All rights reserved Abstract It is problematic to allow reasoning about infinite

### DRAFT. Diagonalization. Chapter 4

Chapter 4 Diagonalization..the relativized P =?NP question has a positive answer for some oracles and a negative answer for other oracles. We feel that this is further evidence of the difficulty of the

### Learning via Finitely Many Queries

Learning via Finitely Many Queries Andrew C. Lee University of Louisiana at Lafayette 1 Abstract This work introduces a new query inference model that can access data and communicate with a teacher by

### ALGORITHMIC INFORMATION THEORY AND UNDECIDABILITY

PANU RAATIKAINEN ALGORITHMIC INFORMATION THEORY AND UNDECIDABILITY 1. INTRODUCTION Algorithmic information theory, or the theory of Kolmogorov complexity, has become an extraordinarily popular theory,

### Classical Propositional Logic

The Language of A Henkin-style Proof for Natural Deduction January 16, 2013 The Language of A Henkin-style Proof for Natural Deduction Logic Logic is the science of inference. Given a body of information,

17 Advanced Undecidability Proofs In this chapter, we will discuss Rice s Theorem in Section 17.1, and the computational history method in Section 17.3. As discussed in Chapter 16, these are two additional

### Applications of Kolmogorov Complexity to Graphs

Applications of Kolmogorov Complexity to Graphs John Hearn Ran Libeskind-Hadas, Advisor Michael Orrison, Reader December 12, 2005 Department of Mathematics Abstract Kolmogorov complexity is a theory based