Lecture 5: Efficient PAC Learning. 1 Consistent Learning: a Bound on Sample Complexity

Size: px
Start display at page:

Download "Lecture 5: Efficient PAC Learning. 1 Consistent Learning: a Bound on Sample Complexity"

Transcription

1 Universität zu Lübeck Institut für Theoretische Informatik Lecture notes on Knowledge-Based and Learning Systems by Maciej Liśkiewicz Lecture 5: Efficient PAC Learning 1 Consistent Learning: a Bound on Sample Complexity Let X be any finite learning domain, let D be any probability distribution over X, and let C (X) be a concept class. Furthermore, we use H to denote any hypothesis space for C. To simplify notation, we use M to denote the cardinality of any set M. Let m N, m 1 ; then we use X m to denote the m -fold Cartesian product of X. Let x X m, then we write x = (x 1,...,x m ). Now, let c C be any concept. The m -sample of c generated by x is denoted by S(c, x) = x 1, c(x 1 ),...,x m, c(x m ). A hypothesis h H is said to be consistent for an m -sample S(c, x) iff h(x i ) = c(x i ) for all 1 i m. A learner is said to be consistent iff for every target concept c and for every hypothesis output h on an m -sample S(c, x) h is consistent for S(c, x). The formal definition of PAC learning has already been presented above. Moreover, we showed the class of all monomials to be PAC learnable. The general idea behind the algorithm given there can be described as follows: (1) Draw a sufficiently large sample from the oracle EX(c, D). (2) Find some h H that is consistent with all the examples drawn. (3) Output h. Therefore, it is only natural to ask whether or not this strategy may be successful in the general finite case, too. Let us assume that we have a consistent learner. Let c C be any concept, and let h be any hypothesis output by the learner on any m -sample S(c, x), where x has been drawn with respect to the unknown probability distribution D. Assume h to be bad, i.e., d(c, h) > ε. Any such hypothesis will not be consistent with m randomly drawn examples unless all examples are drawn outside the symmetric difference of c and h. Hence, the probability that the particular bad hypothesis h survives m examples is at most (1 ε) m. Consequently, the probability that some bad hypothesis survives m examples is at most H (1 ε) m. Furthermore, we want Pr(d(c, h) > ε) < δ. Hence, we must require: H (1 ε) m δ. In this lecture, we consider finite learning domains only.

2 2 Wissensbasierte und lernende Systeme Now, the latter requirement directly allows to lower bound m. Taking the natural logarithm of both sides, we obtain: ln H + m ln(1 ε) lnδ. Therefore, we have: m > lnδ ln H ln(1 ε) Because of (1 1 z )z < e 1 for all z > 0, we additionally obtain: (1 ε) = ((1 ε) 1/ε ) ε < e ε, and thus Putting it all together, we see that m > 1 ε ln(1 ε) < ε. ( ln H + ln 1 ) = 1 H ln δ ε δ. We summarize the insight obtained by the following theorem. Theorem 1 Let X be any finite learning domain, let C (X) be any concept class, and let H be any hypothesis space for C. Then every consistent learner PAC identifies C with respect to H with sample complexity m = 1 H ln ε δ + 1. The latter theorem delivers a first upper bound on the sample complexity needed to achieve efficient PAC learning. However, it does not say anything concerning the problem to compute consistent hypotheses. Clearly, there is a trivial algorithm to achieve this goal. We may just enumerate all hypotheses. Then we may simply search for the first consistent one in the enumeration fixed. Nevertheless, taking into account that H might be huge, this method will usually take too much time. Hence, further effort is necessary to arrive at practical learning algorithms. 2 Efficient PAC Learnability - Definition The latter observation motivates us to strengthen our requirements concerning the efficiency of PAC learning. It might be not enough to bound the number of examples. Additionally, we shall demand the overall running time to be polynomial in the appropriate parameters. Definition 1. A concept class C is said to be efficiently PAC learnable with respect to the hypothesis space H if C is PAC learnable with respect to H, and there exists a PAC learning algorithm A for C that runs in time polynomial in 1/ε, 1/δ, n (the size of an instance in X ), and size (c) for all ε, δ (0, 1) and all c C. Using Theorem 1 we can establish efficient PAC learnability of a couple of important finite concept classes.

3 M. Liśkiewicz, November Example: Efficient Learnability of k-cnf Formulae By k -CNF we denote the class of all conjunctions such that each clause contains at most k literals. The overall number of clauses containing at most k literals is bounded by 2n + (2n) (2n) k < O(n k ). Hence, ln( k-cnf ) = O(n k ). Therefore, we get the following general theorem. Theorem 2 Let k N + be arbitrarily fixed. The class of all concepts describable by a k -CNF formula is efficiently PAC learnable with respect to k -CNF. We leave the proof as an exercise. Next, by k -DNF we denote the class of all disjunctions such that each monomial contains contains at most k literals. Exercise. Prove the following: Let k N + be arbitrarily fixed. The class of all concepts describable by a k -DNF formula is efficiently PAC learnable with respect to k -DNF. 4 Intractability of Learning 3-Term DNF Formulae First of all we define 3-term DNF. Let X = {0, 1} n, n 1 be the Boolean learning domain. Then we use again L n = {x 1, x 1,..., x n, x n } to denote the set of all relevant literals over X. Now, a term is a conjunction of literals from L n. The set of all disjunctions of at most three terms is called 3-term DNF, e.g., x 1 x 3 x 5 x 2 x 4 x 5 x 2 x 3 is a member of 3-term DNF. We are going to study whether or not 3-term DNF is efficiently PAC learnable. What is an appropriate hypothesis space? Well, this is just a crucial problem as we shall see. However, it might be a good idea to try 3-term DNF itself as hypothesis space. Since the sample complexity depends on 3-term DNF we will first check this quantity. As we already know, there are 3 n + 1 monomials over L n. Hence, there are (3 n + 1) 3 many elements in 3-term DNF. Therefore, ln( 3-term DNF ) = O(n). This looks good. Hence, the only remaining problem we have to address is the complexity of finding consistent hypotheses. However, this is easier said than done. Many researches tried to find a polynomial time algorithm for this problem but nobody succeeded until now. Therefore, it seems desirable to provide at least a lower bound for this complexity. Unfortunately, there is no known method to prove nontrivial super-polynomial lower bounds for particular problems. Alternatively, we may try to relate the complexity of finding consistent hypotheses to the complexity of other problems. Good candidates are problems that have been proven to be complete for some complexity class. Whenever dealing with efficient PAC learning the appropriate complexity class is N P. Then, assuming that N P -complete problems cannot be solved efficiently by a randomized algorithm, we get some strong evidence for the non-efficient PAC learnability. That is, unless someone comes up with a proof of N P = RP, we have proved that 3-term DNF is not efficient PAC learnable with respect to 3-term DNF (here RP denotes the class of decision problems solved by polynomial time random Turing machine: for each possible input string, either there are no accepting computations or else at least half of all computations are accepting). Next, we formally define the consistency problem for 3-term DNF. In accordance with the notation introduced above, we use b = (b 1,...,b m ) to denote any m -tuple of Boolean vectors b 1,...,b m {0, 1} n. We start with the following decision problem. Consistency Problem for 3-term DNF

4 4 Wissensbasierte und lernende Systeme Input: m labeled Boolean vectors from {0, 1} n, i.e., an m -sample S(c, b). Output: yes, if there exists a consistent hypothesis h 3-term DNF for S(c, b). no, otherwise. What can be said concerning the complexity of the Consistency Problem for 3-term DNF is provided by our next theorem. Theorem 3 The Consistency Problem for 3-term DNF is N P -complete. Proof. We reduce graph 3-colorability to the Consistency Problem for 3-term DNF. This shows that the Consistency Problem for 3-term DNF is N P -hard. Since the set of all m -samples, m 1, for which there exists a consistent hypothesis is obviously acceptable by a non-deterministic Turing machine, we are done. Graph 3-colorability is a known N P -complete problem defined as follows (cf. Garey and Johnson [3]). Let G = (V, E) be an undirected graph. G is said to be 3-colorable iff there exists a function χ: V {1, 2, 3} such that (i, j) E implies χ(i) χ(j). Let G = (V, E) be any given graph, where without loss of generality V = {1,..., n}. We consider the following reduction. For each vertex i, a positive example b i is created, where b i = u 1 u 2... u n with u i = 0 and u j = 1 for all j i. For each edge (i, j) E, a negative example e ij is created, where e ij = u 1 u 2... u n with u i = u j = 0 and u k = 1 for all k i, j. The resulting sample is denoted by S(G). Since E n(n 1)/2, this reduction is clearly polynomial time computable. It remains to show that the reduction has been defined appropriately. This is done via the following claims. Claim 1. Let G = (V, E) be an undirected graph, and let S(G) be the sample constructed as above. If there exists a hypothesis h 3-term DNF that is consistent with S(G), then G is 3-colorable. Let h be any hypothesis consistent with S(G). Since h 3-term DNF, we may write h = T 1 T 2 T 3. Since h is consistent, we have h(b i ) = 1 for every vertex i V. Moreover, h is a disjunction. Thus, for every vertex i V there must be a term satisfying it. Therefore, we may define the desired mapping χ as follows: χ(i) = min{r T r (b i ) = 1, 1 r 3}. Now, let (i, j) E ; we have to show that χ(i) χ(j). Suppose the converse, i.e., χ(i) = χ(j). Then the examples b i and b j satisfy the same T r. Since (i, j) E we additionally have that e ij = b i b j (taken bitwise). However, b j and e ij differ just in the i th bit. Taking into account that T r (b i ) = T r (b j ) = 1, it is easy to see that neither the literals x i and x i nor the literals x j and x j can be present in T r. Thus, T r (e ij ) = 1, a contradiction to h(e ij ) = 0. This proves Claim 1. Claim 2. Let G = (V, E) be an undirected graph that is 3-colorable. Then there exist a hypothesis h 3-term DNF that is consistent with S(G). Let χ be the mapping assigning the 3 colors to the vertices of G. We define T 1 = and set h = T 1 T 2 T 3. T 2 = T 3 = i, χ(i) 1 i, χ(i) 2 i, χ(i) 3 x i x i x i

5 M. Liśkiewicz, November We have to show that h is consistent with S(G). First, consider any positive example b i. Let χ(i) = r. Then T r satisfies b i. Now, let e ij be any negative example. Since χ assigns to i and j different colors we have χ(i) χ(j). Let χ(i) = r, then T r cannot satisfy e ij since it contains x j. Analogously, if χ(j) = g, the T g cannot satisfy e ij. Hence, if ever, then T y with χ(i) y χ(j) might satisfy e ij. However, this is also impossible, since T y contains both x i and x j. Thus, h(e ij ) = 0, and hence h is consistent. Finally, Theorem 3 has a nice dual version which we include as exercise. Exercise 1. For all n N, n 1, let L n = {x 1, x 1,..., x n x n } be the set of all relevant literals. Furthermore, let k N, k 3, be arbitrarily fixed. By k -term CNF we denote the set of all conjunctions of at most k clauses, where a clause is again any disjunction of elements from L n. Finally, let C(k) = n 1 k-term CNF. Define the consistency problem for C(k) and prove its N P -completeness. As we have seen, the consistency problem for 3-term DNF is N P -complete. However, at first glance it might seem that we have dealt with the wrong question. Whenever studying the PAC learnability of 3-term DNF, all examples drawn are labeled with respect to some target concept. Hence, there is a consistent hypothesis, i.e., at least the target itself. The problem the learner has to solve is to construct a consistent hypothesis. Therefore, we have to investigate how the decision problem and the construction problem are related to one another. This is done by the following theorem. Theorem 4 If there is an algorithm that efficiently PAC learns 3-term DNF with respect to 3-term DNF then N P = RP. Proof. Let A be any algorithm that efficiently PAC learns 3-term DNF with respect to 3-term DNF. Let q be any fixed polynomial such that the running time of A is polynomially bounded in 1/ε, 1/δ, n, and size( c ). Obviously, size( c ) can be upper bounded by 3n, so we have essentially to deal with 1/ε, 1/δ, n. Next we show how to use this algorithm A to decide in random polynomial time the Consistency problem for 3-term DNF. For a formal definition of randomized algorithms and the complexity class RP we refer to Cormen, Leiserson and Rivest [2]. Let S(c, b) be any labeled m -sample, where c is any Boolean concept (not necessarily from 3-term DNF). Next, we choose ε = 1/(m + 1), and an arbitrary small δ > 0, say δ = The choice of ε guarantees that the hypothesis possibly output by A must be consistent with all examples contained in S(c, b). Conceptually, this refers to the probability distribution where each Boolean vector b i from S(c, b) is equally likely, and all other elements have probability zero. That is, D(b i ) = 1/m for all b i from S(c, b), and D(b j ) = 0 for all other Boolean vectors b j {0, 1} n \ {b 1,..., b m }. Hence, if at least one label is not correctly reflected by the hypothesis h possibly output by A, then d(c, h) 1/m > 1/(m + 1). Next, we run A on input ε, δ and the m -sample S(c, b) at most q(1/ε, 1/δ, S(c, b) ) steps. Since every polynomial is time constructible, it is in polynomial time (again in 1/ε, 1/δ, S(c, b) ) decidable whether or not A has already executed at most q(1/ε, 1/δ, S(c, b) ) many steps. Now, we distinguish the following cases. Case 1. A does not stop after having executed q(1/ε, 1/δ, S(c, b) ) many steps, or it stops but does not output any hypothesis. Then, we conclude that there is no consistent hypothesis for S(c, b) in 3-term DNF. If there is really no consistent hypothesis for S(c, b) in 3-term DNF then the output is for sure correct. Now, assume that there is a consistent hypothesis for S(c, b) in 3-term DNF. Hence, there is a concept ĉ 3-term DNF such that c(b i ) = ĉ(b i ) for all 1 i m. Since the Algorithm A is supposed to PAC learn ĉ with respect to every probability distribution, it must do so for D. Hence, with probability 1 δ it has to output a hypothesis h such that d(ĉ, h) = d(c, h) ε. As shown above, the latter inequality forces

6 6 Wissensbasierte und lernende Systeme A to produce a consistent guess. Since A did not produce any guess, it has failed to PAC learn ĉ. However, this failure probability is bounded by δ. Case 2. A stops after having executed at most q(1/ε, 1/δ, S(c, b) ) many steps and outputs a hypothesis h. Obviously, we can decide in time polynomial in the length of h and S(c, b) whether or not h is consistent with S(c, b). In case it is, we know for sure that there exists a consistent hypothesis in 3-term DNF for S(c, b). If h is not consistent, then we can argue as in Case 1. Thus, with probability 1 δ we may conclude that there is no consistent hypothesis for S(c, b) in 3-term DNF. Putting it all together, we have arrived at an algorithm that has the following properties. If there is no consistent hypothesis for S(c, b) in 3-term DNF, then its output is always correct. If there exists a consistent hypothesis for S(c, b) in 3-term DNF then the above algorithm produces with probability δ a wrong answer. Hence, there exists an ε 0 > 0, ε 0 constant such that with probability 1 δ > 1/2 + ε 0 every m sample for which there exists a consistent hypothesis in 3-term DNF is accepted. Thus, we have an RP algorithm for the Consistency problem for 3-term DNF. Finally, since RP N P, and because of the N P -completeness of the Consistency problem for 3-term DNF, we can conclude that RP = N P. The proof provided above is worth to be analyzed a bit further. We strongly recommend to resolve the following exercise. Exercise 2. Prove the following: If there exists a deterministic algorithm A that constructs for every input m -sample S drawn in accordance with some 3-term DNF formula a consistent hypothesis then P = N P provided A has a running time polynomially bounded in S and n. Now, it is only natural to ask whether or not Theorem 4 implies that 3-term DNF are not efficiently PAC learnable at all. We still have the freedom to choose another hypothesis space. As it turns out, a careful choice of the hypothesis space does really change everything. An alternative choice to 3-term DNF are 3-CNF. 3-CNF is the conjunction of clauses that contain at most 3 literals per clause, i.e., 3-CNF is the set of all formulas of the form (l i1 l i2 l i3 ) i where the l ij L n or empty. We leave it as an exercise to show that every 3-term DNF formula is equivalently representable by a 3-CNF formula. The easiest way to see this is to prove that every 3-term DNF formula f = T 1 T 2 T 3 can be rewritten as T 1 T 2 T 3 = x T 1,y T 2,z T 3 (x y z). The converse is not true, i.e., 3-CNF is more expressive than 3-term DNF. Now Theorem 2 allows the following corollary. Corollary 1 The class of all 3-term DNF is efficiently PAC learnable with respect to the hypothesis space space 3-CNF. Again, the latter corollary is easily generalized. Exercise 3. Prove the following: For every constant k 2, the class of all k -term DNF is efficiently PAC learnable with respect to the hypothesis space space k -CNF.

7 M. Liśkiewicz, November References [1] M. Anthony and N. Biggs (1992), Computational Learning Theory, Cambridge University Press, Cambridge. [2] Cormen, T., Leiserson, C. and Rivest, R (1990) Introduction to Algorithms, The MIT Press, Cambridge, MA. [3] Garey, M.R. and Johnson, D.S. (1979), Computers and Intractability: A Guide to the Theory of N P -completeness, Freeman, San Francisco. [4] M.J. Kearns and U.V. Vazirani (1994), An Introduction to Computational Learning Theory, MIT-Press. [5] B.K. Natarajan (1991), Machine Learning, Morgan Kaufmann Publishers Inc. [6] L.G. Valiant (1984), A theory of the learnable, Communications of the ACM 27,

A Tutorial on Computational Learning Theory Presented at Genetic Programming 1997 Stanford University, July 1997

A Tutorial on Computational Learning Theory Presented at Genetic Programming 1997 Stanford University, July 1997 A Tutorial on Computational Learning Theory Presented at Genetic Programming 1997 Stanford University, July 1997 Vasant Honavar Artificial Intelligence Research Laboratory Department of Computer Science

More information

Computational Learning Theory - Hilary Term : Introduction to the PAC Learning Framework

Computational Learning Theory - Hilary Term : Introduction to the PAC Learning Framework Computational Learning Theory - Hilary Term 2018 1 : Introduction to the PAC Learning Framework Lecturer: Varun Kanade 1 What is computational learning theory? Machine learning techniques lie at the heart

More information

Computational Learning Theory

Computational Learning Theory CS 446 Machine Learning Fall 2016 OCT 11, 2016 Computational Learning Theory Professor: Dan Roth Scribe: Ben Zhou, C. Cervantes 1 PAC Learning We want to develop a theory to relate the probability of successful

More information

Computational Learning Theory. Definitions

Computational Learning Theory. Definitions Computational Learning Theory Computational learning theory is interested in theoretical analyses of the following issues. What is needed to learn effectively? Sample complexity. How many examples? Computational

More information

Complexity Theory VU , SS The Polynomial Hierarchy. Reinhard Pichler

Complexity Theory VU , SS The Polynomial Hierarchy. Reinhard Pichler Complexity Theory Complexity Theory VU 181.142, SS 2018 6. The Polynomial Hierarchy Reinhard Pichler Institut für Informationssysteme Arbeitsbereich DBAI Technische Universität Wien 15 May, 2018 Reinhard

More information

Outline. Complexity Theory EXACT TSP. The Class DP. Definition. Problem EXACT TSP. Complexity of EXACT TSP. Proposition VU 181.

Outline. Complexity Theory EXACT TSP. The Class DP. Definition. Problem EXACT TSP. Complexity of EXACT TSP. Proposition VU 181. Complexity Theory Complexity Theory Outline Complexity Theory VU 181.142, SS 2018 6. The Polynomial Hierarchy Reinhard Pichler Institut für Informationssysteme Arbeitsbereich DBAI Technische Universität

More information

Web-Mining Agents Computational Learning Theory

Web-Mining Agents Computational Learning Theory Web-Mining Agents Computational Learning Theory Prof. Dr. Ralf Möller Dr. Özgür Özcep Universität zu Lübeck Institut für Informationssysteme Tanya Braun (Exercise Lab) Computational Learning Theory (Adapted)

More information

Computational Learning Theory

Computational Learning Theory 0. Computational Learning Theory Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch. 7 Acknowledgement: The present slides are an adaptation of slides drawn by T. Mitchell 1. Main Questions

More information

Probably Approximately Correct Learning - III

Probably Approximately Correct Learning - III Probably Approximately Correct Learning - III Prof. Dan A. Simovici UMB Prof. Dan A. Simovici (UMB) Probably Approximately Correct Learning - III 1 / 18 A property of the hypothesis space Aim : a property

More information

About the impossibility to prove P NP or P = NP and the pseudo-randomness in NP

About the impossibility to prove P NP or P = NP and the pseudo-randomness in NP About the impossibility to prove P NP or P = NP and the pseudo-randomness in NP Prof. Marcel Rémon 1 arxiv:0904.0698v3 [cs.cc] 24 Mar 2016 Abstract The relationship between the complexity classes P and

More information

Computational Learning Theory (COLT)

Computational Learning Theory (COLT) Computational Learning Theory (COLT) Goals: Theoretical characterization of 1 Difficulty of machine learning problems Under what conditions is learning possible and impossible? 2 Capabilities of machine

More information

Lecture 25 of 42. PAC Learning, VC Dimension, and Mistake Bounds

Lecture 25 of 42. PAC Learning, VC Dimension, and Mistake Bounds Lecture 25 of 42 PAC Learning, VC Dimension, and Mistake Bounds Thursday, 15 March 2007 William H. Hsu, KSU http://www.kddresearch.org/courses/spring2007/cis732 Readings: Sections 7.4.17.4.3, 7.5.17.5.3,

More information

Online Learning, Mistake Bounds, Perceptron Algorithm

Online Learning, Mistake Bounds, Perceptron Algorithm Online Learning, Mistake Bounds, Perceptron Algorithm 1 Online Learning So far the focus of the course has been on batch learning, where algorithms are presented with a sample of training data, from which

More information

2 P vs. NP and Diagonalization

2 P vs. NP and Diagonalization 2 P vs NP and Diagonalization CS 6810 Theory of Computing, Fall 2012 Instructor: David Steurer (sc2392) Date: 08/28/2012 In this lecture, we cover the following topics: 1 3SAT is NP hard; 2 Time hierarchies;

More information

Show that the following problems are NP-complete

Show that the following problems are NP-complete Show that the following problems are NP-complete April 7, 2018 Below is a list of 30 exercises in which you are asked to prove that some problem is NP-complete. The goal is to better understand the theory

More information

Chapter 2. Reductions and NP. 2.1 Reductions Continued The Satisfiability Problem (SAT) SAT 3SAT. CS 573: Algorithms, Fall 2013 August 29, 2013

Chapter 2. Reductions and NP. 2.1 Reductions Continued The Satisfiability Problem (SAT) SAT 3SAT. CS 573: Algorithms, Fall 2013 August 29, 2013 Chapter 2 Reductions and NP CS 573: Algorithms, Fall 2013 August 29, 2013 2.1 Reductions Continued 2.1.1 The Satisfiability Problem SAT 2.1.1.1 Propositional Formulas Definition 2.1.1. Consider a set of

More information

CS 151 Complexity Theory Spring Solution Set 5

CS 151 Complexity Theory Spring Solution Set 5 CS 151 Complexity Theory Spring 2017 Solution Set 5 Posted: May 17 Chris Umans 1. We are given a Boolean circuit C on n variables x 1, x 2,..., x n with m, and gates. Our 3-CNF formula will have m auxiliary

More information

Introduction to Algorithms / Algorithms I Lecturer: Michael Dinitz Topic: Intro to Learning Theory Date: 12/8/16

Introduction to Algorithms / Algorithms I Lecturer: Michael Dinitz Topic: Intro to Learning Theory Date: 12/8/16 600.463 Introduction to Algorithms / Algorithms I Lecturer: Michael Dinitz Topic: Intro to Learning Theory Date: 12/8/16 25.1 Introduction Today we re going to talk about machine learning, but from an

More information

1 More finite deterministic automata

1 More finite deterministic automata CS 125 Section #6 Finite automata October 18, 2016 1 More finite deterministic automata Exercise. Consider the following game with two players: Repeatedly flip a coin. On heads, player 1 gets a point.

More information

PAC Learning. prof. dr Arno Siebes. Algorithmic Data Analysis Group Department of Information and Computing Sciences Universiteit Utrecht

PAC Learning. prof. dr Arno Siebes. Algorithmic Data Analysis Group Department of Information and Computing Sciences Universiteit Utrecht PAC Learning prof. dr Arno Siebes Algorithmic Data Analysis Group Department of Information and Computing Sciences Universiteit Utrecht Recall: PAC Learning (Version 1) A hypothesis class H is PAC learnable

More information

Lecture Notes 4. Issued 8 March 2018

Lecture Notes 4. Issued 8 March 2018 CM30073 Advanced Algorithms and Complexity 1. Structure of the class NP Lecture Notes 4 Issued 8 March 2018 Recall that it is not known whether or not P = NP, the widely accepted hypothesis being that

More information

Computational Learning Theory

Computational Learning Theory Computational Learning Theory Slides by and Nathalie Japkowicz (Reading: R&N AIMA 3 rd ed., Chapter 18.5) Computational Learning Theory Inductive learning: given the training set, a learning algorithm

More information

Design and Analysis of Algorithms

Design and Analysis of Algorithms Design and Analysis of Algorithms CSE 5311 Lecture 25 NP Completeness Junzhou Huang, Ph.D. Department of Computer Science and Engineering CSE5311 Design and Analysis of Algorithms 1 NP-Completeness Some

More information

Theory of Computer Science. Theory of Computer Science. E4.1 Overview E4.2 3SAT. E4.3 Graph Problems. E4.4 Summary.

Theory of Computer Science. Theory of Computer Science. E4.1 Overview E4.2 3SAT. E4.3 Graph Problems. E4.4 Summary. Theory of Computer Science May 30, 2016 E4. Some NP-Complete Problems, Part I Theory of Computer Science E4. Some NP-Complete Problems, Part I Malte Helmert University of Basel May 30, 2016 E4.1 Overview

More information

Hierarchical Concept Learning

Hierarchical Concept Learning COMS 6998-4 Fall 2017 Octorber 30, 2017 Hierarchical Concept Learning Presenter: Xuefeng Hu Scribe: Qinyao He 1 Introduction It has been shown that learning arbitrary polynomial-size circuits is computationally

More information

Lecture 2. 1 More N P-Compete Languages. Notes on Complexity Theory: Fall 2005 Last updated: September, Jonathan Katz

Lecture 2. 1 More N P-Compete Languages. Notes on Complexity Theory: Fall 2005 Last updated: September, Jonathan Katz Notes on Complexity Theory: Fall 2005 Last updated: September, 2005 Jonathan Katz Lecture 2 1 More N P-Compete Languages It will be nice to find more natural N P-complete languages. To that end, we ine

More information

Computational learning theory. PAC learning. VC dimension.

Computational learning theory. PAC learning. VC dimension. Computational learning theory. PAC learning. VC dimension. Petr Pošík Czech Technical University in Prague Faculty of Electrical Engineering Dept. of Cybernetics COLT 2 Concept...........................................................................................................

More information

CS 5114: Theory of Algorithms. Tractable Problems. Tractable Problems (cont) Decision Problems. Clifford A. Shaffer. Spring 2014

CS 5114: Theory of Algorithms. Tractable Problems. Tractable Problems (cont) Decision Problems. Clifford A. Shaffer. Spring 2014 Department of Computer Science Virginia Tech Blacksburg, Virginia Copyright c 2014 by Clifford A. Shaffer : Theory of Algorithms Title page : Theory of Algorithms Clifford A. Shaffer Spring 2014 Clifford

More information

1.1 P, NP, and NP-complete

1.1 P, NP, and NP-complete CSC5160: Combinatorial Optimization and Approximation Algorithms Topic: Introduction to NP-complete Problems Date: 11/01/2008 Lecturer: Lap Chi Lau Scribe: Jerry Jilin Le This lecture gives a general introduction

More information

Limits to Approximability: When Algorithms Won't Help You. Note: Contents of today s lecture won t be on the exam

Limits to Approximability: When Algorithms Won't Help You. Note: Contents of today s lecture won t be on the exam Limits to Approximability: When Algorithms Won't Help You Note: Contents of today s lecture won t be on the exam Outline Limits to Approximability: basic results Detour: Provers, verifiers, and NP Graph

More information

CS 6375: Machine Learning Computational Learning Theory

CS 6375: Machine Learning Computational Learning Theory CS 6375: Machine Learning Computational Learning Theory Vibhav Gogate The University of Texas at Dallas Many slides borrowed from Ray Mooney 1 Learning Theory Theoretical characterizations of Difficulty

More information

Notes on Computer Theory Last updated: November, Circuits

Notes on Computer Theory Last updated: November, Circuits Notes on Computer Theory Last updated: November, 2015 Circuits Notes by Jonathan Katz, lightly edited by Dov Gordon. 1 Circuits Boolean circuits offer an alternate model of computation: a non-uniform one

More information

Harvard CS 121 and CSCI E-121 Lecture 22: The P vs. NP Question and NP-completeness

Harvard CS 121 and CSCI E-121 Lecture 22: The P vs. NP Question and NP-completeness Harvard CS 121 and CSCI E-121 Lecture 22: The P vs. NP Question and NP-completeness Harry Lewis November 19, 2013 Reading: Sipser 7.4, 7.5. For culture : Computers and Intractability: A Guide to the Theory

More information

Foundations of Machine Learning and Data Science. Lecturer: Avrim Blum Lecture 9: October 7, 2015

Foundations of Machine Learning and Data Science. Lecturer: Avrim Blum Lecture 9: October 7, 2015 10-806 Foundations of Machine Learning and Data Science Lecturer: Avrim Blum Lecture 9: October 7, 2015 1 Computational Hardness of Learning Today we will talk about some computational hardness results

More information

Handout 5. α a1 a n. }, where. xi if a i = 1 1 if a i = 0.

Handout 5. α a1 a n. }, where. xi if a i = 1 1 if a i = 0. Notes on Complexity Theory Last updated: October, 2005 Jonathan Katz Handout 5 1 An Improved Upper-Bound on Circuit Size Here we show the result promised in the previous lecture regarding an upper-bound

More information

an efficient procedure for the decision problem. We illustrate this phenomenon for the Satisfiability problem.

an efficient procedure for the decision problem. We illustrate this phenomenon for the Satisfiability problem. 1 More on NP In this set of lecture notes, we examine the class NP in more detail. We give a characterization of NP which justifies the guess and verify paradigm, and study the complexity of solving search

More information

Introduction to Computational Learning Theory

Introduction to Computational Learning Theory Introduction to Computational Learning Theory The classification problem Consistent Hypothesis Model Probably Approximately Correct (PAC) Learning c Hung Q. Ngo (SUNY at Buffalo) CSE 694 A Fun Course 1

More information

Essential facts about NP-completeness:

Essential facts about NP-completeness: CMPSCI611: NP Completeness Lecture 17 Essential facts about NP-completeness: Any NP-complete problem can be solved by a simple, but exponentially slow algorithm. We don t have polynomial-time solutions

More information

On the Sample Complexity of Noise-Tolerant Learning

On the Sample Complexity of Noise-Tolerant Learning On the Sample Complexity of Noise-Tolerant Learning Javed A. Aslam Department of Computer Science Dartmouth College Hanover, NH 03755 Scott E. Decatur Laboratory for Computer Science Massachusetts Institute

More information

Classes of Boolean Functions

Classes of Boolean Functions Classes of Boolean Functions Nader H. Bshouty Eyal Kushilevitz Abstract Here we give classes of Boolean functions that considered in COLT. Classes of Functions Here we introduce the basic classes of functions

More information

Critical Reading of Optimization Methods for Logical Inference [1]

Critical Reading of Optimization Methods for Logical Inference [1] Critical Reading of Optimization Methods for Logical Inference [1] Undergraduate Research Internship Department of Management Sciences Fall 2007 Supervisor: Dr. Miguel Anjos UNIVERSITY OF WATERLOO Rajesh

More information

NP Complete Problems. COMP 215 Lecture 20

NP Complete Problems. COMP 215 Lecture 20 NP Complete Problems COMP 215 Lecture 20 Complexity Theory Complexity theory is a research area unto itself. The central project is classifying problems as either tractable or intractable. Tractable Worst

More information

Introduction to Machine Learning

Introduction to Machine Learning Introduction to Machine Learning PAC Learning and VC Dimension Varun Chandola Computer Science & Engineering State University of New York at Buffalo Buffalo, NY, USA chandola@buffalo.edu Chandola@UB CSE

More information

NP-Completeness. Until now we have been designing algorithms for specific problems

NP-Completeness. Until now we have been designing algorithms for specific problems NP-Completeness 1 Introduction Until now we have been designing algorithms for specific problems We have seen running times O(log n), O(n), O(n log n), O(n 2 ), O(n 3 )... We have also discussed lower

More information

CS Lecture 29 P, NP, and NP-Completeness. k ) for all k. Fall The class P. The class NP

CS Lecture 29 P, NP, and NP-Completeness. k ) for all k. Fall The class P. The class NP CS 301 - Lecture 29 P, NP, and NP-Completeness Fall 2008 Review Languages and Grammars Alphabets, strings, languages Regular Languages Deterministic Finite and Nondeterministic Automata Equivalence of

More information

Learning Large-Alphabet and Analog Circuits with Value Injection Queries

Learning Large-Alphabet and Analog Circuits with Value Injection Queries Learning Large-Alphabet and Analog Circuits with Value Injection Queries Dana Angluin 1 James Aspnes 1, Jiang Chen 2, Lev Reyzin 1,, 1 Computer Science Department, Yale University {angluin,aspnes}@cs.yale.edu,

More information

NP-Completeness. Andreas Klappenecker. [based on slides by Prof. Welch]

NP-Completeness. Andreas Klappenecker. [based on slides by Prof. Welch] NP-Completeness Andreas Klappenecker [based on slides by Prof. Welch] 1 Prelude: Informal Discussion (Incidentally, we will never get very formal in this course) 2 Polynomial Time Algorithms Most of the

More information

Computational Learning Theory. CS534 - Machine Learning

Computational Learning Theory. CS534 - Machine Learning Computational Learning Theory CS534 Machine Learning Introduction Computational learning theory Provides a theoretical analysis of learning Shows when a learning algorithm can be expected to succeed Shows

More information

CS 5114: Theory of Algorithms

CS 5114: Theory of Algorithms CS 5114: Theory of Algorithms Clifford A. Shaffer Department of Computer Science Virginia Tech Blacksburg, Virginia Spring 2014 Copyright c 2014 by Clifford A. Shaffer CS 5114: Theory of Algorithms Spring

More information

Week 3: Reductions and Completeness

Week 3: Reductions and Completeness Computational Complexity Theory Summer HSSP 2018 Week 3: Reductions and Completeness Dylan Hendrickson MIT Educational Studies Program 3.1 Reductions Suppose I know how to solve some problem quickly. How

More information

Computational Learning Theory: PAC Model

Computational Learning Theory: PAC Model Computational Learning Theory: PAC Model Subhash Suri May 19, 2015 1 A rectangle Learning Game These notes are based on the paper A Theory of the Learnable by Valiant, the book by Kearns-Vazirani, and

More information

6.045: Automata, Computability, and Complexity (GITCS) Class 15 Nancy Lynch

6.045: Automata, Computability, and Complexity (GITCS) Class 15 Nancy Lynch 6.045: Automata, Computability, and Complexity (GITCS) Class 15 Nancy Lynch Today: More Complexity Theory Polynomial-time reducibility, NP-completeness, and the Satisfiability (SAT) problem Topics: Introduction

More information

Lecture 7: Passive Learning

Lecture 7: Passive Learning CS 880: Advanced Complexity Theory 2/8/2008 Lecture 7: Passive Learning Instructor: Dieter van Melkebeek Scribe: Tom Watson In the previous lectures, we studied harmonic analysis as a tool for analyzing

More information

A An Overview of Complexity Theory for the Algorithm Designer

A An Overview of Complexity Theory for the Algorithm Designer A An Overview of Complexity Theory for the Algorithm Designer A.1 Certificates and the class NP A decision problem is one whose answer is either yes or no. Two examples are: SAT: Given a Boolean formula

More information

Lecture 3. 1 Terminology. 2 Non-Deterministic Space Complexity. Notes on Complexity Theory: Fall 2005 Last updated: September, 2005.

Lecture 3. 1 Terminology. 2 Non-Deterministic Space Complexity. Notes on Complexity Theory: Fall 2005 Last updated: September, 2005. Notes on Complexity Theory: Fall 2005 Last updated: September, 2005 Jonathan Katz Lecture 3 1 Terminology For any complexity class C, we define the class coc as follows: coc def = { L L C }. One class

More information

UC Berkeley CS 170: Efficient Algorithms and Intractable Problems Handout 22 Lecturer: David Wagner April 24, Notes 22 for CS 170

UC Berkeley CS 170: Efficient Algorithms and Intractable Problems Handout 22 Lecturer: David Wagner April 24, Notes 22 for CS 170 UC Berkeley CS 170: Efficient Algorithms and Intractable Problems Handout 22 Lecturer: David Wagner April 24, 2003 Notes 22 for CS 170 1 NP-completeness of Circuit-SAT We will prove that the circuit satisfiability

More information

Computational and Statistical Learning Theory

Computational and Statistical Learning Theory Computational and Statistical Learning Theory TTIC 31120 Prof. Nati Srebro Lecture 6: Computational Complexity of Learning Proper vs Improper Learning Efficient PAC Learning Definition: A family H n of

More information

Discriminative Learning can Succeed where Generative Learning Fails

Discriminative Learning can Succeed where Generative Learning Fails Discriminative Learning can Succeed where Generative Learning Fails Philip M. Long, a Rocco A. Servedio, b,,1 Hans Ulrich Simon c a Google, Mountain View, CA, USA b Columbia University, New York, New York,

More information

NP-Complete Problems. More reductions

NP-Complete Problems. More reductions NP-Complete Problems More reductions Definitions P: problems that can be solved in polynomial time (typically in n, size of input) on a deterministic Turing machine Any normal computer simulates a DTM

More information

Can PAC Learning Algorithms Tolerate. Random Attribute Noise? Sally A. Goldman. Department of Computer Science. Washington University

Can PAC Learning Algorithms Tolerate. Random Attribute Noise? Sally A. Goldman. Department of Computer Science. Washington University Can PAC Learning Algorithms Tolerate Random Attribute Noise? Sally A. Goldman Department of Computer Science Washington University St. Louis, Missouri 63130 Robert H. Sloan y Dept. of Electrical Engineering

More information

Lecture 1 : Probabilistic Method

Lecture 1 : Probabilistic Method IITM-CS6845: Theory Jan 04, 01 Lecturer: N.S.Narayanaswamy Lecture 1 : Probabilistic Method Scribe: R.Krithika The probabilistic method is a technique to deal with combinatorial problems by introducing

More information

P is the class of problems for which there are algorithms that solve the problem in time O(n k ) for some constant k.

P is the class of problems for which there are algorithms that solve the problem in time O(n k ) for some constant k. Complexity Theory Problems are divided into complexity classes. Informally: So far in this course, almost all algorithms had polynomial running time, i.e., on inputs of size n, worst-case running time

More information

CS 395T Computational Learning Theory. Scribe: Mike Halcrow. x 4. x 2. x 6

CS 395T Computational Learning Theory. Scribe: Mike Halcrow. x 4. x 2. x 6 CS 395T Computational Learning Theory Lecture 3: September 0, 2007 Lecturer: Adam Klivans Scribe: Mike Halcrow 3. Decision List Recap In the last class, we determined that, when learning a t-decision list,

More information

Computational Learning Theory

Computational Learning Theory Computational Learning Theory Sinh Hoa Nguyen, Hung Son Nguyen Polish-Japanese Institute of Information Technology Institute of Mathematics, Warsaw University February 14, 2006 inh Hoa Nguyen, Hung Son

More information

Polynomial time Prediction Strategy with almost Optimal Mistake Probability

Polynomial time Prediction Strategy with almost Optimal Mistake Probability Polynomial time Prediction Strategy with almost Optimal Mistake Probability Nader H. Bshouty Department of Computer Science Technion, 32000 Haifa, Israel bshouty@cs.technion.ac.il Abstract We give the

More information

Lecture 29: Computational Learning Theory

Lecture 29: Computational Learning Theory CS 710: Complexity Theory 5/4/2010 Lecture 29: Computational Learning Theory Instructor: Dieter van Melkebeek Scribe: Dmitri Svetlov and Jake Rosin Today we will provide a brief introduction to computational

More information

NP Completeness and Approximation Algorithms

NP Completeness and Approximation Algorithms Winter School on Optimization Techniques December 15-20, 2016 Organized by ACMU, ISI and IEEE CEDA NP Completeness and Approximation Algorithms Susmita Sur-Kolay Advanced Computing and Microelectronic

More information

Theory of Computation CS3102 Spring 2014 A tale of computers, math, problem solving, life, love and tragic death

Theory of Computation CS3102 Spring 2014 A tale of computers, math, problem solving, life, love and tragic death Theory of Computation CS3102 Spring 2014 A tale of computers, math, problem solving, life, love and tragic death Nathan Brunelle Department of Computer Science University of Virginia www.cs.virginia.edu/~njb2b/theory

More information

Proving SAT does not have Small Circuits with an Application to the Two Queries Problem

Proving SAT does not have Small Circuits with an Application to the Two Queries Problem Proving SAT does not have Small Circuits with an Application to the Two Queries Problem Lance Fortnow A. Pavan Samik Sengupta Abstract We show that if SAT does not have small circuits, then there must

More information

10.1 The Formal Model

10.1 The Formal Model 67577 Intro. to Machine Learning Fall semester, 2008/9 Lecture 10: The Formal (PAC) Learning Model Lecturer: Amnon Shashua Scribe: Amnon Shashua 1 We have see so far algorithms that explicitly estimate

More information

1 PSPACE-Completeness

1 PSPACE-Completeness CS 6743 Lecture 14 1 Fall 2007 1 PSPACE-Completeness Recall the NP-complete problem SAT: Is a given Boolean formula φ(x 1,..., x n ) satisfiable? The same question can be stated equivalently as: Is the

More information

Lecture Notes Each circuit agrees with M on inputs of length equal to its index, i.e. n, x {0, 1} n, C n (x) = M(x).

Lecture Notes Each circuit agrees with M on inputs of length equal to its index, i.e. n, x {0, 1} n, C n (x) = M(x). CS 221: Computational Complexity Prof. Salil Vadhan Lecture Notes 4 February 3, 2010 Scribe: Jonathan Pines 1 Agenda P-/NP- Completeness NP-intermediate problems NP vs. co-np L, NL 2 Recap Last time, we

More information

Summer School on Introduction to Algorithms and Optimization Techniques July 4-12, 2017 Organized by ACMU, ISI and IEEE CEDA.

Summer School on Introduction to Algorithms and Optimization Techniques July 4-12, 2017 Organized by ACMU, ISI and IEEE CEDA. Summer School on Introduction to Algorithms and Optimization Techniques July 4-12, 2017 Organized by ACMU, ISI and IEEE CEDA NP Completeness Susmita Sur-Kolay Advanced Computing and Microelectronics Unit

More information

COL 352 Introduction to Automata and Theory of Computation Major Exam, Sem II , Max 80, Time 2 hr. Name Entry No. Group

COL 352 Introduction to Automata and Theory of Computation Major Exam, Sem II , Max 80, Time 2 hr. Name Entry No. Group COL 352 Introduction to Automata and Theory of Computation Major Exam, Sem II 2015-16, Max 80, Time 2 hr Name Entry No. Group Note (i) Write your answers neatly and precisely in the space provided with

More information

SAT, NP, NP-Completeness

SAT, NP, NP-Completeness CS 473: Algorithms, Spring 2018 SAT, NP, NP-Completeness Lecture 22 April 13, 2018 Most slides are courtesy Prof. Chekuri Ruta (UIUC) CS473 1 Spring 2018 1 / 57 Part I Reductions Continued Ruta (UIUC)

More information

Comp487/587 - Boolean Formulas

Comp487/587 - Boolean Formulas Comp487/587 - Boolean Formulas 1 Logic and SAT 1.1 What is a Boolean Formula Logic is a way through which we can analyze and reason about simple or complicated events. In particular, we are interested

More information

Limitations of Algorithm Power

Limitations of Algorithm Power Limitations of Algorithm Power Objectives We now move into the third and final major theme for this course. 1. Tools for analyzing algorithms. 2. Design strategies for designing algorithms. 3. Identifying

More information

CSCI3390-Second Test with Solutions

CSCI3390-Second Test with Solutions CSCI3390-Second Test with Solutions April 26, 2016 Each of the 15 parts of the problems below is worth 10 points, except for the more involved 4(d), which is worth 20. A perfect score is 100: if your score

More information

1 Distributional problems

1 Distributional problems CSCI 5170: Computational Complexity Lecture 6 The Chinese University of Hong Kong, Spring 2016 23 February 2016 The theory of NP-completeness has been applied to explain why brute-force search is essentially

More information

Apropos of an errata in ÜB 10 exercise 3

Apropos of an errata in ÜB 10 exercise 3 Apropos of an errata in ÜB 10 exercise 3 Komplexität von Algorithmen SS13 The last exercise of the last exercise sheet was incorrectly formulated and could not be properly solved. Since no one spotted

More information

NP-Completeness. f(n) \ n n sec sec sec. n sec 24.3 sec 5.2 mins. 2 n sec 17.9 mins 35.

NP-Completeness. f(n) \ n n sec sec sec. n sec 24.3 sec 5.2 mins. 2 n sec 17.9 mins 35. NP-Completeness Reference: Computers and Intractability: A Guide to the Theory of NP-Completeness by Garey and Johnson, W.H. Freeman and Company, 1979. NP-Completeness 1 General Problems, Input Size and

More information

Machine Learning. Computational Learning Theory. Eric Xing , Fall Lecture 9, October 5, 2016

Machine Learning. Computational Learning Theory. Eric Xing , Fall Lecture 9, October 5, 2016 Machine Learning 10-701, Fall 2016 Computational Learning Theory Eric Xing Lecture 9, October 5, 2016 Reading: Chap. 7 T.M book Eric Xing @ CMU, 2006-2016 1 Generalizability of Learning In machine learning

More information

Lecture 4. 1 Circuit Complexity. Notes on Complexity Theory: Fall 2005 Last updated: September, Jonathan Katz

Lecture 4. 1 Circuit Complexity. Notes on Complexity Theory: Fall 2005 Last updated: September, Jonathan Katz Notes on Complexity Theory: Fall 2005 Last updated: September, 2005 Jonathan Katz Lecture 4 1 Circuit Complexity Circuits are directed, acyclic graphs where nodes are called gates and edges are called

More information

THE VAPNIK- CHERVONENKIS DIMENSION and LEARNABILITY

THE VAPNIK- CHERVONENKIS DIMENSION and LEARNABILITY THE VAPNIK- CHERVONENKIS DIMENSION and LEARNABILITY Dan A. Simovici UMB, Doctoral Summer School Iasi, Romania What is Machine Learning? The Vapnik-Chervonenkis Dimension Probabilistic Learning Potential

More information

Computational Complexity and Intractability: An Introduction to the Theory of NP. Chapter 9

Computational Complexity and Intractability: An Introduction to the Theory of NP. Chapter 9 1 Computational Complexity and Intractability: An Introduction to the Theory of NP Chapter 9 2 Objectives Classify problems as tractable or intractable Define decision problems Define the class P Define

More information

A Lower Bound of 2 n Conditional Jumps for Boolean Satisfiability on A Random Access Machine

A Lower Bound of 2 n Conditional Jumps for Boolean Satisfiability on A Random Access Machine A Lower Bound of 2 n Conditional Jumps for Boolean Satisfiability on A Random Access Machine Samuel C. Hsieh Computer Science Department, Ball State University July 3, 2014 Abstract We establish a lower

More information

Outline. Complexity Theory. Introduction. What is abduction? Motivation. Reference VU , SS Logic-Based Abduction

Outline. Complexity Theory. Introduction. What is abduction? Motivation. Reference VU , SS Logic-Based Abduction Complexity Theory Complexity Theory Outline Complexity Theory VU 181.142, SS 2018 7. Logic-Based Abduction Reinhard Pichler Institut für Informationssysteme Arbeitsbereich DBAI Technische Universität Wien

More information

1 The Probably Approximately Correct (PAC) Model

1 The Probably Approximately Correct (PAC) Model COS 511: Theoretical Machine Learning Lecturer: Rob Schapire Lecture #3 Scribe: Yuhui Luo February 11, 2008 1 The Probably Approximately Correct (PAC) Model A target concept class C is PAC-learnable by

More information

Statistical and Computational Learning Theory

Statistical and Computational Learning Theory Statistical and Computational Learning Theory Fundamental Question: Predict Error Rates Given: Find: The space H of hypotheses The number and distribution of the training examples S The complexity of the

More information

Computational Learning Theory for Artificial Neural Networks

Computational Learning Theory for Artificial Neural Networks Computational Learning Theory for Artificial Neural Networks Martin Anthony and Norman Biggs Department of Statistical and Mathematical Sciences, London School of Economics and Political Science, Houghton

More information

1 Computational Problems

1 Computational Problems Stanford University CS254: Computational Complexity Handout 2 Luca Trevisan March 31, 2010 Last revised 4/29/2010 In this lecture we define NP, we state the P versus NP problem, we prove that its formulation

More information

P P P NP-Hard: L is NP-hard if for all L NP, L L. Thus, if we could solve L in polynomial. Cook's Theorem and Reductions

P P P NP-Hard: L is NP-hard if for all L NP, L L. Thus, if we could solve L in polynomial. Cook's Theorem and Reductions Summary of the previous lecture Recall that we mentioned the following topics: P: is the set of decision problems (or languages) that are solvable in polynomial time. NP: is the set of decision problems

More information

Decentralized Control of Discrete Event Systems with Bounded or Unbounded Delay Communication

Decentralized Control of Discrete Event Systems with Bounded or Unbounded Delay Communication Decentralized Control of Discrete Event Systems with Bounded or Unbounded Delay Communication Stavros Tripakis Abstract We introduce problems of decentralized control with communication, where we explicitly

More information

Computational Learning Theory

Computational Learning Theory 1 Computational Learning Theory 2 Computational learning theory Introduction Is it possible to identify classes of learning problems that are inherently easy or difficult? Can we characterize the number

More information

20.1 2SAT. CS125 Lecture 20 Fall 2016

20.1 2SAT. CS125 Lecture 20 Fall 2016 CS125 Lecture 20 Fall 2016 20.1 2SAT We show yet another possible way to solve the 2SAT problem. Recall that the input to 2SAT is a logical expression that is the conunction (AND) of a set of clauses,

More information

Propositional Resolution

Propositional Resolution Artificial Intelligence Propositional Resolution Marco Piastra Propositional Resolution 1] Deductive systems and automation Is problem decidible? A deductive system a la Hilbert (i.e. derivation using

More information

ICML '97 and AAAI '97 Tutorials

ICML '97 and AAAI '97 Tutorials A Short Course in Computational Learning Theory: ICML '97 and AAAI '97 Tutorials Michael Kearns AT&T Laboratories Outline Sample Complexity/Learning Curves: nite classes, Occam's VC dimension Razor, Best

More information

CS154, Lecture 15: Cook-Levin Theorem SAT, 3SAT

CS154, Lecture 15: Cook-Levin Theorem SAT, 3SAT CS154, Lecture 15: Cook-Levin Theorem SAT, 3SAT Definition: A language B is NP-complete if: 1. B NP 2. Every A in NP is poly-time reducible to B That is, A P B When this is true, we say B is NP-hard On

More information

CMSC 858F: Algorithmic Lower Bounds Fall SAT and NP-Hardness

CMSC 858F: Algorithmic Lower Bounds Fall SAT and NP-Hardness CMSC 858F: Algorithmic Lower Bounds Fall 2014 3-SAT and NP-Hardness Instructor: Mohammad T. Hajiaghayi Scribe: Philip Dasler September 23, 2014 The most important NP-Complete (logic) problem family! 1

More information