Lecture 5: Efficient PAC Learning. 1 Consistent Learning: a Bound on Sample Complexity
|
|
- Kellie Cole
- 5 years ago
- Views:
Transcription
1 Universität zu Lübeck Institut für Theoretische Informatik Lecture notes on Knowledge-Based and Learning Systems by Maciej Liśkiewicz Lecture 5: Efficient PAC Learning 1 Consistent Learning: a Bound on Sample Complexity Let X be any finite learning domain, let D be any probability distribution over X, and let C (X) be a concept class. Furthermore, we use H to denote any hypothesis space for C. To simplify notation, we use M to denote the cardinality of any set M. Let m N, m 1 ; then we use X m to denote the m -fold Cartesian product of X. Let x X m, then we write x = (x 1,...,x m ). Now, let c C be any concept. The m -sample of c generated by x is denoted by S(c, x) = x 1, c(x 1 ),...,x m, c(x m ). A hypothesis h H is said to be consistent for an m -sample S(c, x) iff h(x i ) = c(x i ) for all 1 i m. A learner is said to be consistent iff for every target concept c and for every hypothesis output h on an m -sample S(c, x) h is consistent for S(c, x). The formal definition of PAC learning has already been presented above. Moreover, we showed the class of all monomials to be PAC learnable. The general idea behind the algorithm given there can be described as follows: (1) Draw a sufficiently large sample from the oracle EX(c, D). (2) Find some h H that is consistent with all the examples drawn. (3) Output h. Therefore, it is only natural to ask whether or not this strategy may be successful in the general finite case, too. Let us assume that we have a consistent learner. Let c C be any concept, and let h be any hypothesis output by the learner on any m -sample S(c, x), where x has been drawn with respect to the unknown probability distribution D. Assume h to be bad, i.e., d(c, h) > ε. Any such hypothesis will not be consistent with m randomly drawn examples unless all examples are drawn outside the symmetric difference of c and h. Hence, the probability that the particular bad hypothesis h survives m examples is at most (1 ε) m. Consequently, the probability that some bad hypothesis survives m examples is at most H (1 ε) m. Furthermore, we want Pr(d(c, h) > ε) < δ. Hence, we must require: H (1 ε) m δ. In this lecture, we consider finite learning domains only.
2 2 Wissensbasierte und lernende Systeme Now, the latter requirement directly allows to lower bound m. Taking the natural logarithm of both sides, we obtain: ln H + m ln(1 ε) lnδ. Therefore, we have: m > lnδ ln H ln(1 ε) Because of (1 1 z )z < e 1 for all z > 0, we additionally obtain: (1 ε) = ((1 ε) 1/ε ) ε < e ε, and thus Putting it all together, we see that m > 1 ε ln(1 ε) < ε. ( ln H + ln 1 ) = 1 H ln δ ε δ. We summarize the insight obtained by the following theorem. Theorem 1 Let X be any finite learning domain, let C (X) be any concept class, and let H be any hypothesis space for C. Then every consistent learner PAC identifies C with respect to H with sample complexity m = 1 H ln ε δ + 1. The latter theorem delivers a first upper bound on the sample complexity needed to achieve efficient PAC learning. However, it does not say anything concerning the problem to compute consistent hypotheses. Clearly, there is a trivial algorithm to achieve this goal. We may just enumerate all hypotheses. Then we may simply search for the first consistent one in the enumeration fixed. Nevertheless, taking into account that H might be huge, this method will usually take too much time. Hence, further effort is necessary to arrive at practical learning algorithms. 2 Efficient PAC Learnability - Definition The latter observation motivates us to strengthen our requirements concerning the efficiency of PAC learning. It might be not enough to bound the number of examples. Additionally, we shall demand the overall running time to be polynomial in the appropriate parameters. Definition 1. A concept class C is said to be efficiently PAC learnable with respect to the hypothesis space H if C is PAC learnable with respect to H, and there exists a PAC learning algorithm A for C that runs in time polynomial in 1/ε, 1/δ, n (the size of an instance in X ), and size (c) for all ε, δ (0, 1) and all c C. Using Theorem 1 we can establish efficient PAC learnability of a couple of important finite concept classes.
3 M. Liśkiewicz, November Example: Efficient Learnability of k-cnf Formulae By k -CNF we denote the class of all conjunctions such that each clause contains at most k literals. The overall number of clauses containing at most k literals is bounded by 2n + (2n) (2n) k < O(n k ). Hence, ln( k-cnf ) = O(n k ). Therefore, we get the following general theorem. Theorem 2 Let k N + be arbitrarily fixed. The class of all concepts describable by a k -CNF formula is efficiently PAC learnable with respect to k -CNF. We leave the proof as an exercise. Next, by k -DNF we denote the class of all disjunctions such that each monomial contains contains at most k literals. Exercise. Prove the following: Let k N + be arbitrarily fixed. The class of all concepts describable by a k -DNF formula is efficiently PAC learnable with respect to k -DNF. 4 Intractability of Learning 3-Term DNF Formulae First of all we define 3-term DNF. Let X = {0, 1} n, n 1 be the Boolean learning domain. Then we use again L n = {x 1, x 1,..., x n, x n } to denote the set of all relevant literals over X. Now, a term is a conjunction of literals from L n. The set of all disjunctions of at most three terms is called 3-term DNF, e.g., x 1 x 3 x 5 x 2 x 4 x 5 x 2 x 3 is a member of 3-term DNF. We are going to study whether or not 3-term DNF is efficiently PAC learnable. What is an appropriate hypothesis space? Well, this is just a crucial problem as we shall see. However, it might be a good idea to try 3-term DNF itself as hypothesis space. Since the sample complexity depends on 3-term DNF we will first check this quantity. As we already know, there are 3 n + 1 monomials over L n. Hence, there are (3 n + 1) 3 many elements in 3-term DNF. Therefore, ln( 3-term DNF ) = O(n). This looks good. Hence, the only remaining problem we have to address is the complexity of finding consistent hypotheses. However, this is easier said than done. Many researches tried to find a polynomial time algorithm for this problem but nobody succeeded until now. Therefore, it seems desirable to provide at least a lower bound for this complexity. Unfortunately, there is no known method to prove nontrivial super-polynomial lower bounds for particular problems. Alternatively, we may try to relate the complexity of finding consistent hypotheses to the complexity of other problems. Good candidates are problems that have been proven to be complete for some complexity class. Whenever dealing with efficient PAC learning the appropriate complexity class is N P. Then, assuming that N P -complete problems cannot be solved efficiently by a randomized algorithm, we get some strong evidence for the non-efficient PAC learnability. That is, unless someone comes up with a proof of N P = RP, we have proved that 3-term DNF is not efficient PAC learnable with respect to 3-term DNF (here RP denotes the class of decision problems solved by polynomial time random Turing machine: for each possible input string, either there are no accepting computations or else at least half of all computations are accepting). Next, we formally define the consistency problem for 3-term DNF. In accordance with the notation introduced above, we use b = (b 1,...,b m ) to denote any m -tuple of Boolean vectors b 1,...,b m {0, 1} n. We start with the following decision problem. Consistency Problem for 3-term DNF
4 4 Wissensbasierte und lernende Systeme Input: m labeled Boolean vectors from {0, 1} n, i.e., an m -sample S(c, b). Output: yes, if there exists a consistent hypothesis h 3-term DNF for S(c, b). no, otherwise. What can be said concerning the complexity of the Consistency Problem for 3-term DNF is provided by our next theorem. Theorem 3 The Consistency Problem for 3-term DNF is N P -complete. Proof. We reduce graph 3-colorability to the Consistency Problem for 3-term DNF. This shows that the Consistency Problem for 3-term DNF is N P -hard. Since the set of all m -samples, m 1, for which there exists a consistent hypothesis is obviously acceptable by a non-deterministic Turing machine, we are done. Graph 3-colorability is a known N P -complete problem defined as follows (cf. Garey and Johnson [3]). Let G = (V, E) be an undirected graph. G is said to be 3-colorable iff there exists a function χ: V {1, 2, 3} such that (i, j) E implies χ(i) χ(j). Let G = (V, E) be any given graph, where without loss of generality V = {1,..., n}. We consider the following reduction. For each vertex i, a positive example b i is created, where b i = u 1 u 2... u n with u i = 0 and u j = 1 for all j i. For each edge (i, j) E, a negative example e ij is created, where e ij = u 1 u 2... u n with u i = u j = 0 and u k = 1 for all k i, j. The resulting sample is denoted by S(G). Since E n(n 1)/2, this reduction is clearly polynomial time computable. It remains to show that the reduction has been defined appropriately. This is done via the following claims. Claim 1. Let G = (V, E) be an undirected graph, and let S(G) be the sample constructed as above. If there exists a hypothesis h 3-term DNF that is consistent with S(G), then G is 3-colorable. Let h be any hypothesis consistent with S(G). Since h 3-term DNF, we may write h = T 1 T 2 T 3. Since h is consistent, we have h(b i ) = 1 for every vertex i V. Moreover, h is a disjunction. Thus, for every vertex i V there must be a term satisfying it. Therefore, we may define the desired mapping χ as follows: χ(i) = min{r T r (b i ) = 1, 1 r 3}. Now, let (i, j) E ; we have to show that χ(i) χ(j). Suppose the converse, i.e., χ(i) = χ(j). Then the examples b i and b j satisfy the same T r. Since (i, j) E we additionally have that e ij = b i b j (taken bitwise). However, b j and e ij differ just in the i th bit. Taking into account that T r (b i ) = T r (b j ) = 1, it is easy to see that neither the literals x i and x i nor the literals x j and x j can be present in T r. Thus, T r (e ij ) = 1, a contradiction to h(e ij ) = 0. This proves Claim 1. Claim 2. Let G = (V, E) be an undirected graph that is 3-colorable. Then there exist a hypothesis h 3-term DNF that is consistent with S(G). Let χ be the mapping assigning the 3 colors to the vertices of G. We define T 1 = and set h = T 1 T 2 T 3. T 2 = T 3 = i, χ(i) 1 i, χ(i) 2 i, χ(i) 3 x i x i x i
5 M. Liśkiewicz, November We have to show that h is consistent with S(G). First, consider any positive example b i. Let χ(i) = r. Then T r satisfies b i. Now, let e ij be any negative example. Since χ assigns to i and j different colors we have χ(i) χ(j). Let χ(i) = r, then T r cannot satisfy e ij since it contains x j. Analogously, if χ(j) = g, the T g cannot satisfy e ij. Hence, if ever, then T y with χ(i) y χ(j) might satisfy e ij. However, this is also impossible, since T y contains both x i and x j. Thus, h(e ij ) = 0, and hence h is consistent. Finally, Theorem 3 has a nice dual version which we include as exercise. Exercise 1. For all n N, n 1, let L n = {x 1, x 1,..., x n x n } be the set of all relevant literals. Furthermore, let k N, k 3, be arbitrarily fixed. By k -term CNF we denote the set of all conjunctions of at most k clauses, where a clause is again any disjunction of elements from L n. Finally, let C(k) = n 1 k-term CNF. Define the consistency problem for C(k) and prove its N P -completeness. As we have seen, the consistency problem for 3-term DNF is N P -complete. However, at first glance it might seem that we have dealt with the wrong question. Whenever studying the PAC learnability of 3-term DNF, all examples drawn are labeled with respect to some target concept. Hence, there is a consistent hypothesis, i.e., at least the target itself. The problem the learner has to solve is to construct a consistent hypothesis. Therefore, we have to investigate how the decision problem and the construction problem are related to one another. This is done by the following theorem. Theorem 4 If there is an algorithm that efficiently PAC learns 3-term DNF with respect to 3-term DNF then N P = RP. Proof. Let A be any algorithm that efficiently PAC learns 3-term DNF with respect to 3-term DNF. Let q be any fixed polynomial such that the running time of A is polynomially bounded in 1/ε, 1/δ, n, and size( c ). Obviously, size( c ) can be upper bounded by 3n, so we have essentially to deal with 1/ε, 1/δ, n. Next we show how to use this algorithm A to decide in random polynomial time the Consistency problem for 3-term DNF. For a formal definition of randomized algorithms and the complexity class RP we refer to Cormen, Leiserson and Rivest [2]. Let S(c, b) be any labeled m -sample, where c is any Boolean concept (not necessarily from 3-term DNF). Next, we choose ε = 1/(m + 1), and an arbitrary small δ > 0, say δ = The choice of ε guarantees that the hypothesis possibly output by A must be consistent with all examples contained in S(c, b). Conceptually, this refers to the probability distribution where each Boolean vector b i from S(c, b) is equally likely, and all other elements have probability zero. That is, D(b i ) = 1/m for all b i from S(c, b), and D(b j ) = 0 for all other Boolean vectors b j {0, 1} n \ {b 1,..., b m }. Hence, if at least one label is not correctly reflected by the hypothesis h possibly output by A, then d(c, h) 1/m > 1/(m + 1). Next, we run A on input ε, δ and the m -sample S(c, b) at most q(1/ε, 1/δ, S(c, b) ) steps. Since every polynomial is time constructible, it is in polynomial time (again in 1/ε, 1/δ, S(c, b) ) decidable whether or not A has already executed at most q(1/ε, 1/δ, S(c, b) ) many steps. Now, we distinguish the following cases. Case 1. A does not stop after having executed q(1/ε, 1/δ, S(c, b) ) many steps, or it stops but does not output any hypothesis. Then, we conclude that there is no consistent hypothesis for S(c, b) in 3-term DNF. If there is really no consistent hypothesis for S(c, b) in 3-term DNF then the output is for sure correct. Now, assume that there is a consistent hypothesis for S(c, b) in 3-term DNF. Hence, there is a concept ĉ 3-term DNF such that c(b i ) = ĉ(b i ) for all 1 i m. Since the Algorithm A is supposed to PAC learn ĉ with respect to every probability distribution, it must do so for D. Hence, with probability 1 δ it has to output a hypothesis h such that d(ĉ, h) = d(c, h) ε. As shown above, the latter inequality forces
6 6 Wissensbasierte und lernende Systeme A to produce a consistent guess. Since A did not produce any guess, it has failed to PAC learn ĉ. However, this failure probability is bounded by δ. Case 2. A stops after having executed at most q(1/ε, 1/δ, S(c, b) ) many steps and outputs a hypothesis h. Obviously, we can decide in time polynomial in the length of h and S(c, b) whether or not h is consistent with S(c, b). In case it is, we know for sure that there exists a consistent hypothesis in 3-term DNF for S(c, b). If h is not consistent, then we can argue as in Case 1. Thus, with probability 1 δ we may conclude that there is no consistent hypothesis for S(c, b) in 3-term DNF. Putting it all together, we have arrived at an algorithm that has the following properties. If there is no consistent hypothesis for S(c, b) in 3-term DNF, then its output is always correct. If there exists a consistent hypothesis for S(c, b) in 3-term DNF then the above algorithm produces with probability δ a wrong answer. Hence, there exists an ε 0 > 0, ε 0 constant such that with probability 1 δ > 1/2 + ε 0 every m sample for which there exists a consistent hypothesis in 3-term DNF is accepted. Thus, we have an RP algorithm for the Consistency problem for 3-term DNF. Finally, since RP N P, and because of the N P -completeness of the Consistency problem for 3-term DNF, we can conclude that RP = N P. The proof provided above is worth to be analyzed a bit further. We strongly recommend to resolve the following exercise. Exercise 2. Prove the following: If there exists a deterministic algorithm A that constructs for every input m -sample S drawn in accordance with some 3-term DNF formula a consistent hypothesis then P = N P provided A has a running time polynomially bounded in S and n. Now, it is only natural to ask whether or not Theorem 4 implies that 3-term DNF are not efficiently PAC learnable at all. We still have the freedom to choose another hypothesis space. As it turns out, a careful choice of the hypothesis space does really change everything. An alternative choice to 3-term DNF are 3-CNF. 3-CNF is the conjunction of clauses that contain at most 3 literals per clause, i.e., 3-CNF is the set of all formulas of the form (l i1 l i2 l i3 ) i where the l ij L n or empty. We leave it as an exercise to show that every 3-term DNF formula is equivalently representable by a 3-CNF formula. The easiest way to see this is to prove that every 3-term DNF formula f = T 1 T 2 T 3 can be rewritten as T 1 T 2 T 3 = x T 1,y T 2,z T 3 (x y z). The converse is not true, i.e., 3-CNF is more expressive than 3-term DNF. Now Theorem 2 allows the following corollary. Corollary 1 The class of all 3-term DNF is efficiently PAC learnable with respect to the hypothesis space space 3-CNF. Again, the latter corollary is easily generalized. Exercise 3. Prove the following: For every constant k 2, the class of all k -term DNF is efficiently PAC learnable with respect to the hypothesis space space k -CNF.
7 M. Liśkiewicz, November References [1] M. Anthony and N. Biggs (1992), Computational Learning Theory, Cambridge University Press, Cambridge. [2] Cormen, T., Leiserson, C. and Rivest, R (1990) Introduction to Algorithms, The MIT Press, Cambridge, MA. [3] Garey, M.R. and Johnson, D.S. (1979), Computers and Intractability: A Guide to the Theory of N P -completeness, Freeman, San Francisco. [4] M.J. Kearns and U.V. Vazirani (1994), An Introduction to Computational Learning Theory, MIT-Press. [5] B.K. Natarajan (1991), Machine Learning, Morgan Kaufmann Publishers Inc. [6] L.G. Valiant (1984), A theory of the learnable, Communications of the ACM 27,
A Tutorial on Computational Learning Theory Presented at Genetic Programming 1997 Stanford University, July 1997
A Tutorial on Computational Learning Theory Presented at Genetic Programming 1997 Stanford University, July 1997 Vasant Honavar Artificial Intelligence Research Laboratory Department of Computer Science
More informationComputational Learning Theory - Hilary Term : Introduction to the PAC Learning Framework
Computational Learning Theory - Hilary Term 2018 1 : Introduction to the PAC Learning Framework Lecturer: Varun Kanade 1 What is computational learning theory? Machine learning techniques lie at the heart
More informationComputational Learning Theory
CS 446 Machine Learning Fall 2016 OCT 11, 2016 Computational Learning Theory Professor: Dan Roth Scribe: Ben Zhou, C. Cervantes 1 PAC Learning We want to develop a theory to relate the probability of successful
More informationComputational Learning Theory. Definitions
Computational Learning Theory Computational learning theory is interested in theoretical analyses of the following issues. What is needed to learn effectively? Sample complexity. How many examples? Computational
More informationComplexity Theory VU , SS The Polynomial Hierarchy. Reinhard Pichler
Complexity Theory Complexity Theory VU 181.142, SS 2018 6. The Polynomial Hierarchy Reinhard Pichler Institut für Informationssysteme Arbeitsbereich DBAI Technische Universität Wien 15 May, 2018 Reinhard
More informationOutline. Complexity Theory EXACT TSP. The Class DP. Definition. Problem EXACT TSP. Complexity of EXACT TSP. Proposition VU 181.
Complexity Theory Complexity Theory Outline Complexity Theory VU 181.142, SS 2018 6. The Polynomial Hierarchy Reinhard Pichler Institut für Informationssysteme Arbeitsbereich DBAI Technische Universität
More informationWeb-Mining Agents Computational Learning Theory
Web-Mining Agents Computational Learning Theory Prof. Dr. Ralf Möller Dr. Özgür Özcep Universität zu Lübeck Institut für Informationssysteme Tanya Braun (Exercise Lab) Computational Learning Theory (Adapted)
More informationComputational Learning Theory
0. Computational Learning Theory Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch. 7 Acknowledgement: The present slides are an adaptation of slides drawn by T. Mitchell 1. Main Questions
More informationProbably Approximately Correct Learning - III
Probably Approximately Correct Learning - III Prof. Dan A. Simovici UMB Prof. Dan A. Simovici (UMB) Probably Approximately Correct Learning - III 1 / 18 A property of the hypothesis space Aim : a property
More informationAbout the impossibility to prove P NP or P = NP and the pseudo-randomness in NP
About the impossibility to prove P NP or P = NP and the pseudo-randomness in NP Prof. Marcel Rémon 1 arxiv:0904.0698v3 [cs.cc] 24 Mar 2016 Abstract The relationship between the complexity classes P and
More informationComputational Learning Theory (COLT)
Computational Learning Theory (COLT) Goals: Theoretical characterization of 1 Difficulty of machine learning problems Under what conditions is learning possible and impossible? 2 Capabilities of machine
More informationLecture 25 of 42. PAC Learning, VC Dimension, and Mistake Bounds
Lecture 25 of 42 PAC Learning, VC Dimension, and Mistake Bounds Thursday, 15 March 2007 William H. Hsu, KSU http://www.kddresearch.org/courses/spring2007/cis732 Readings: Sections 7.4.17.4.3, 7.5.17.5.3,
More informationOnline Learning, Mistake Bounds, Perceptron Algorithm
Online Learning, Mistake Bounds, Perceptron Algorithm 1 Online Learning So far the focus of the course has been on batch learning, where algorithms are presented with a sample of training data, from which
More information2 P vs. NP and Diagonalization
2 P vs NP and Diagonalization CS 6810 Theory of Computing, Fall 2012 Instructor: David Steurer (sc2392) Date: 08/28/2012 In this lecture, we cover the following topics: 1 3SAT is NP hard; 2 Time hierarchies;
More informationShow that the following problems are NP-complete
Show that the following problems are NP-complete April 7, 2018 Below is a list of 30 exercises in which you are asked to prove that some problem is NP-complete. The goal is to better understand the theory
More informationChapter 2. Reductions and NP. 2.1 Reductions Continued The Satisfiability Problem (SAT) SAT 3SAT. CS 573: Algorithms, Fall 2013 August 29, 2013
Chapter 2 Reductions and NP CS 573: Algorithms, Fall 2013 August 29, 2013 2.1 Reductions Continued 2.1.1 The Satisfiability Problem SAT 2.1.1.1 Propositional Formulas Definition 2.1.1. Consider a set of
More informationCS 151 Complexity Theory Spring Solution Set 5
CS 151 Complexity Theory Spring 2017 Solution Set 5 Posted: May 17 Chris Umans 1. We are given a Boolean circuit C on n variables x 1, x 2,..., x n with m, and gates. Our 3-CNF formula will have m auxiliary
More informationIntroduction to Algorithms / Algorithms I Lecturer: Michael Dinitz Topic: Intro to Learning Theory Date: 12/8/16
600.463 Introduction to Algorithms / Algorithms I Lecturer: Michael Dinitz Topic: Intro to Learning Theory Date: 12/8/16 25.1 Introduction Today we re going to talk about machine learning, but from an
More information1 More finite deterministic automata
CS 125 Section #6 Finite automata October 18, 2016 1 More finite deterministic automata Exercise. Consider the following game with two players: Repeatedly flip a coin. On heads, player 1 gets a point.
More informationPAC Learning. prof. dr Arno Siebes. Algorithmic Data Analysis Group Department of Information and Computing Sciences Universiteit Utrecht
PAC Learning prof. dr Arno Siebes Algorithmic Data Analysis Group Department of Information and Computing Sciences Universiteit Utrecht Recall: PAC Learning (Version 1) A hypothesis class H is PAC learnable
More informationLecture Notes 4. Issued 8 March 2018
CM30073 Advanced Algorithms and Complexity 1. Structure of the class NP Lecture Notes 4 Issued 8 March 2018 Recall that it is not known whether or not P = NP, the widely accepted hypothesis being that
More informationComputational Learning Theory
Computational Learning Theory Slides by and Nathalie Japkowicz (Reading: R&N AIMA 3 rd ed., Chapter 18.5) Computational Learning Theory Inductive learning: given the training set, a learning algorithm
More informationDesign and Analysis of Algorithms
Design and Analysis of Algorithms CSE 5311 Lecture 25 NP Completeness Junzhou Huang, Ph.D. Department of Computer Science and Engineering CSE5311 Design and Analysis of Algorithms 1 NP-Completeness Some
More informationTheory of Computer Science. Theory of Computer Science. E4.1 Overview E4.2 3SAT. E4.3 Graph Problems. E4.4 Summary.
Theory of Computer Science May 30, 2016 E4. Some NP-Complete Problems, Part I Theory of Computer Science E4. Some NP-Complete Problems, Part I Malte Helmert University of Basel May 30, 2016 E4.1 Overview
More informationHierarchical Concept Learning
COMS 6998-4 Fall 2017 Octorber 30, 2017 Hierarchical Concept Learning Presenter: Xuefeng Hu Scribe: Qinyao He 1 Introduction It has been shown that learning arbitrary polynomial-size circuits is computationally
More informationLecture 2. 1 More N P-Compete Languages. Notes on Complexity Theory: Fall 2005 Last updated: September, Jonathan Katz
Notes on Complexity Theory: Fall 2005 Last updated: September, 2005 Jonathan Katz Lecture 2 1 More N P-Compete Languages It will be nice to find more natural N P-complete languages. To that end, we ine
More informationComputational learning theory. PAC learning. VC dimension.
Computational learning theory. PAC learning. VC dimension. Petr Pošík Czech Technical University in Prague Faculty of Electrical Engineering Dept. of Cybernetics COLT 2 Concept...........................................................................................................
More informationCS 5114: Theory of Algorithms. Tractable Problems. Tractable Problems (cont) Decision Problems. Clifford A. Shaffer. Spring 2014
Department of Computer Science Virginia Tech Blacksburg, Virginia Copyright c 2014 by Clifford A. Shaffer : Theory of Algorithms Title page : Theory of Algorithms Clifford A. Shaffer Spring 2014 Clifford
More information1.1 P, NP, and NP-complete
CSC5160: Combinatorial Optimization and Approximation Algorithms Topic: Introduction to NP-complete Problems Date: 11/01/2008 Lecturer: Lap Chi Lau Scribe: Jerry Jilin Le This lecture gives a general introduction
More informationLimits to Approximability: When Algorithms Won't Help You. Note: Contents of today s lecture won t be on the exam
Limits to Approximability: When Algorithms Won't Help You Note: Contents of today s lecture won t be on the exam Outline Limits to Approximability: basic results Detour: Provers, verifiers, and NP Graph
More informationCS 6375: Machine Learning Computational Learning Theory
CS 6375: Machine Learning Computational Learning Theory Vibhav Gogate The University of Texas at Dallas Many slides borrowed from Ray Mooney 1 Learning Theory Theoretical characterizations of Difficulty
More informationNotes on Computer Theory Last updated: November, Circuits
Notes on Computer Theory Last updated: November, 2015 Circuits Notes by Jonathan Katz, lightly edited by Dov Gordon. 1 Circuits Boolean circuits offer an alternate model of computation: a non-uniform one
More informationHarvard CS 121 and CSCI E-121 Lecture 22: The P vs. NP Question and NP-completeness
Harvard CS 121 and CSCI E-121 Lecture 22: The P vs. NP Question and NP-completeness Harry Lewis November 19, 2013 Reading: Sipser 7.4, 7.5. For culture : Computers and Intractability: A Guide to the Theory
More informationFoundations of Machine Learning and Data Science. Lecturer: Avrim Blum Lecture 9: October 7, 2015
10-806 Foundations of Machine Learning and Data Science Lecturer: Avrim Blum Lecture 9: October 7, 2015 1 Computational Hardness of Learning Today we will talk about some computational hardness results
More informationHandout 5. α a1 a n. }, where. xi if a i = 1 1 if a i = 0.
Notes on Complexity Theory Last updated: October, 2005 Jonathan Katz Handout 5 1 An Improved Upper-Bound on Circuit Size Here we show the result promised in the previous lecture regarding an upper-bound
More informationan efficient procedure for the decision problem. We illustrate this phenomenon for the Satisfiability problem.
1 More on NP In this set of lecture notes, we examine the class NP in more detail. We give a characterization of NP which justifies the guess and verify paradigm, and study the complexity of solving search
More informationIntroduction to Computational Learning Theory
Introduction to Computational Learning Theory The classification problem Consistent Hypothesis Model Probably Approximately Correct (PAC) Learning c Hung Q. Ngo (SUNY at Buffalo) CSE 694 A Fun Course 1
More informationEssential facts about NP-completeness:
CMPSCI611: NP Completeness Lecture 17 Essential facts about NP-completeness: Any NP-complete problem can be solved by a simple, but exponentially slow algorithm. We don t have polynomial-time solutions
More informationOn the Sample Complexity of Noise-Tolerant Learning
On the Sample Complexity of Noise-Tolerant Learning Javed A. Aslam Department of Computer Science Dartmouth College Hanover, NH 03755 Scott E. Decatur Laboratory for Computer Science Massachusetts Institute
More informationClasses of Boolean Functions
Classes of Boolean Functions Nader H. Bshouty Eyal Kushilevitz Abstract Here we give classes of Boolean functions that considered in COLT. Classes of Functions Here we introduce the basic classes of functions
More informationCritical Reading of Optimization Methods for Logical Inference [1]
Critical Reading of Optimization Methods for Logical Inference [1] Undergraduate Research Internship Department of Management Sciences Fall 2007 Supervisor: Dr. Miguel Anjos UNIVERSITY OF WATERLOO Rajesh
More informationNP Complete Problems. COMP 215 Lecture 20
NP Complete Problems COMP 215 Lecture 20 Complexity Theory Complexity theory is a research area unto itself. The central project is classifying problems as either tractable or intractable. Tractable Worst
More informationIntroduction to Machine Learning
Introduction to Machine Learning PAC Learning and VC Dimension Varun Chandola Computer Science & Engineering State University of New York at Buffalo Buffalo, NY, USA chandola@buffalo.edu Chandola@UB CSE
More informationNP-Completeness. Until now we have been designing algorithms for specific problems
NP-Completeness 1 Introduction Until now we have been designing algorithms for specific problems We have seen running times O(log n), O(n), O(n log n), O(n 2 ), O(n 3 )... We have also discussed lower
More informationCS Lecture 29 P, NP, and NP-Completeness. k ) for all k. Fall The class P. The class NP
CS 301 - Lecture 29 P, NP, and NP-Completeness Fall 2008 Review Languages and Grammars Alphabets, strings, languages Regular Languages Deterministic Finite and Nondeterministic Automata Equivalence of
More informationLearning Large-Alphabet and Analog Circuits with Value Injection Queries
Learning Large-Alphabet and Analog Circuits with Value Injection Queries Dana Angluin 1 James Aspnes 1, Jiang Chen 2, Lev Reyzin 1,, 1 Computer Science Department, Yale University {angluin,aspnes}@cs.yale.edu,
More informationNP-Completeness. Andreas Klappenecker. [based on slides by Prof. Welch]
NP-Completeness Andreas Klappenecker [based on slides by Prof. Welch] 1 Prelude: Informal Discussion (Incidentally, we will never get very formal in this course) 2 Polynomial Time Algorithms Most of the
More informationComputational Learning Theory. CS534 - Machine Learning
Computational Learning Theory CS534 Machine Learning Introduction Computational learning theory Provides a theoretical analysis of learning Shows when a learning algorithm can be expected to succeed Shows
More informationCS 5114: Theory of Algorithms
CS 5114: Theory of Algorithms Clifford A. Shaffer Department of Computer Science Virginia Tech Blacksburg, Virginia Spring 2014 Copyright c 2014 by Clifford A. Shaffer CS 5114: Theory of Algorithms Spring
More informationWeek 3: Reductions and Completeness
Computational Complexity Theory Summer HSSP 2018 Week 3: Reductions and Completeness Dylan Hendrickson MIT Educational Studies Program 3.1 Reductions Suppose I know how to solve some problem quickly. How
More informationComputational Learning Theory: PAC Model
Computational Learning Theory: PAC Model Subhash Suri May 19, 2015 1 A rectangle Learning Game These notes are based on the paper A Theory of the Learnable by Valiant, the book by Kearns-Vazirani, and
More information6.045: Automata, Computability, and Complexity (GITCS) Class 15 Nancy Lynch
6.045: Automata, Computability, and Complexity (GITCS) Class 15 Nancy Lynch Today: More Complexity Theory Polynomial-time reducibility, NP-completeness, and the Satisfiability (SAT) problem Topics: Introduction
More informationLecture 7: Passive Learning
CS 880: Advanced Complexity Theory 2/8/2008 Lecture 7: Passive Learning Instructor: Dieter van Melkebeek Scribe: Tom Watson In the previous lectures, we studied harmonic analysis as a tool for analyzing
More informationA An Overview of Complexity Theory for the Algorithm Designer
A An Overview of Complexity Theory for the Algorithm Designer A.1 Certificates and the class NP A decision problem is one whose answer is either yes or no. Two examples are: SAT: Given a Boolean formula
More informationLecture 3. 1 Terminology. 2 Non-Deterministic Space Complexity. Notes on Complexity Theory: Fall 2005 Last updated: September, 2005.
Notes on Complexity Theory: Fall 2005 Last updated: September, 2005 Jonathan Katz Lecture 3 1 Terminology For any complexity class C, we define the class coc as follows: coc def = { L L C }. One class
More informationUC Berkeley CS 170: Efficient Algorithms and Intractable Problems Handout 22 Lecturer: David Wagner April 24, Notes 22 for CS 170
UC Berkeley CS 170: Efficient Algorithms and Intractable Problems Handout 22 Lecturer: David Wagner April 24, 2003 Notes 22 for CS 170 1 NP-completeness of Circuit-SAT We will prove that the circuit satisfiability
More informationComputational and Statistical Learning Theory
Computational and Statistical Learning Theory TTIC 31120 Prof. Nati Srebro Lecture 6: Computational Complexity of Learning Proper vs Improper Learning Efficient PAC Learning Definition: A family H n of
More informationDiscriminative Learning can Succeed where Generative Learning Fails
Discriminative Learning can Succeed where Generative Learning Fails Philip M. Long, a Rocco A. Servedio, b,,1 Hans Ulrich Simon c a Google, Mountain View, CA, USA b Columbia University, New York, New York,
More informationNP-Complete Problems. More reductions
NP-Complete Problems More reductions Definitions P: problems that can be solved in polynomial time (typically in n, size of input) on a deterministic Turing machine Any normal computer simulates a DTM
More informationCan PAC Learning Algorithms Tolerate. Random Attribute Noise? Sally A. Goldman. Department of Computer Science. Washington University
Can PAC Learning Algorithms Tolerate Random Attribute Noise? Sally A. Goldman Department of Computer Science Washington University St. Louis, Missouri 63130 Robert H. Sloan y Dept. of Electrical Engineering
More informationLecture 1 : Probabilistic Method
IITM-CS6845: Theory Jan 04, 01 Lecturer: N.S.Narayanaswamy Lecture 1 : Probabilistic Method Scribe: R.Krithika The probabilistic method is a technique to deal with combinatorial problems by introducing
More informationP is the class of problems for which there are algorithms that solve the problem in time O(n k ) for some constant k.
Complexity Theory Problems are divided into complexity classes. Informally: So far in this course, almost all algorithms had polynomial running time, i.e., on inputs of size n, worst-case running time
More informationCS 395T Computational Learning Theory. Scribe: Mike Halcrow. x 4. x 2. x 6
CS 395T Computational Learning Theory Lecture 3: September 0, 2007 Lecturer: Adam Klivans Scribe: Mike Halcrow 3. Decision List Recap In the last class, we determined that, when learning a t-decision list,
More informationComputational Learning Theory
Computational Learning Theory Sinh Hoa Nguyen, Hung Son Nguyen Polish-Japanese Institute of Information Technology Institute of Mathematics, Warsaw University February 14, 2006 inh Hoa Nguyen, Hung Son
More informationPolynomial time Prediction Strategy with almost Optimal Mistake Probability
Polynomial time Prediction Strategy with almost Optimal Mistake Probability Nader H. Bshouty Department of Computer Science Technion, 32000 Haifa, Israel bshouty@cs.technion.ac.il Abstract We give the
More informationLecture 29: Computational Learning Theory
CS 710: Complexity Theory 5/4/2010 Lecture 29: Computational Learning Theory Instructor: Dieter van Melkebeek Scribe: Dmitri Svetlov and Jake Rosin Today we will provide a brief introduction to computational
More informationNP Completeness and Approximation Algorithms
Winter School on Optimization Techniques December 15-20, 2016 Organized by ACMU, ISI and IEEE CEDA NP Completeness and Approximation Algorithms Susmita Sur-Kolay Advanced Computing and Microelectronic
More informationTheory of Computation CS3102 Spring 2014 A tale of computers, math, problem solving, life, love and tragic death
Theory of Computation CS3102 Spring 2014 A tale of computers, math, problem solving, life, love and tragic death Nathan Brunelle Department of Computer Science University of Virginia www.cs.virginia.edu/~njb2b/theory
More informationProving SAT does not have Small Circuits with an Application to the Two Queries Problem
Proving SAT does not have Small Circuits with an Application to the Two Queries Problem Lance Fortnow A. Pavan Samik Sengupta Abstract We show that if SAT does not have small circuits, then there must
More information10.1 The Formal Model
67577 Intro. to Machine Learning Fall semester, 2008/9 Lecture 10: The Formal (PAC) Learning Model Lecturer: Amnon Shashua Scribe: Amnon Shashua 1 We have see so far algorithms that explicitly estimate
More information1 PSPACE-Completeness
CS 6743 Lecture 14 1 Fall 2007 1 PSPACE-Completeness Recall the NP-complete problem SAT: Is a given Boolean formula φ(x 1,..., x n ) satisfiable? The same question can be stated equivalently as: Is the
More informationLecture Notes Each circuit agrees with M on inputs of length equal to its index, i.e. n, x {0, 1} n, C n (x) = M(x).
CS 221: Computational Complexity Prof. Salil Vadhan Lecture Notes 4 February 3, 2010 Scribe: Jonathan Pines 1 Agenda P-/NP- Completeness NP-intermediate problems NP vs. co-np L, NL 2 Recap Last time, we
More informationSummer School on Introduction to Algorithms and Optimization Techniques July 4-12, 2017 Organized by ACMU, ISI and IEEE CEDA.
Summer School on Introduction to Algorithms and Optimization Techniques July 4-12, 2017 Organized by ACMU, ISI and IEEE CEDA NP Completeness Susmita Sur-Kolay Advanced Computing and Microelectronics Unit
More informationCOL 352 Introduction to Automata and Theory of Computation Major Exam, Sem II , Max 80, Time 2 hr. Name Entry No. Group
COL 352 Introduction to Automata and Theory of Computation Major Exam, Sem II 2015-16, Max 80, Time 2 hr Name Entry No. Group Note (i) Write your answers neatly and precisely in the space provided with
More informationSAT, NP, NP-Completeness
CS 473: Algorithms, Spring 2018 SAT, NP, NP-Completeness Lecture 22 April 13, 2018 Most slides are courtesy Prof. Chekuri Ruta (UIUC) CS473 1 Spring 2018 1 / 57 Part I Reductions Continued Ruta (UIUC)
More informationComp487/587 - Boolean Formulas
Comp487/587 - Boolean Formulas 1 Logic and SAT 1.1 What is a Boolean Formula Logic is a way through which we can analyze and reason about simple or complicated events. In particular, we are interested
More informationLimitations of Algorithm Power
Limitations of Algorithm Power Objectives We now move into the third and final major theme for this course. 1. Tools for analyzing algorithms. 2. Design strategies for designing algorithms. 3. Identifying
More informationCSCI3390-Second Test with Solutions
CSCI3390-Second Test with Solutions April 26, 2016 Each of the 15 parts of the problems below is worth 10 points, except for the more involved 4(d), which is worth 20. A perfect score is 100: if your score
More information1 Distributional problems
CSCI 5170: Computational Complexity Lecture 6 The Chinese University of Hong Kong, Spring 2016 23 February 2016 The theory of NP-completeness has been applied to explain why brute-force search is essentially
More informationApropos of an errata in ÜB 10 exercise 3
Apropos of an errata in ÜB 10 exercise 3 Komplexität von Algorithmen SS13 The last exercise of the last exercise sheet was incorrectly formulated and could not be properly solved. Since no one spotted
More informationNP-Completeness. f(n) \ n n sec sec sec. n sec 24.3 sec 5.2 mins. 2 n sec 17.9 mins 35.
NP-Completeness Reference: Computers and Intractability: A Guide to the Theory of NP-Completeness by Garey and Johnson, W.H. Freeman and Company, 1979. NP-Completeness 1 General Problems, Input Size and
More informationMachine Learning. Computational Learning Theory. Eric Xing , Fall Lecture 9, October 5, 2016
Machine Learning 10-701, Fall 2016 Computational Learning Theory Eric Xing Lecture 9, October 5, 2016 Reading: Chap. 7 T.M book Eric Xing @ CMU, 2006-2016 1 Generalizability of Learning In machine learning
More informationLecture 4. 1 Circuit Complexity. Notes on Complexity Theory: Fall 2005 Last updated: September, Jonathan Katz
Notes on Complexity Theory: Fall 2005 Last updated: September, 2005 Jonathan Katz Lecture 4 1 Circuit Complexity Circuits are directed, acyclic graphs where nodes are called gates and edges are called
More informationTHE VAPNIK- CHERVONENKIS DIMENSION and LEARNABILITY
THE VAPNIK- CHERVONENKIS DIMENSION and LEARNABILITY Dan A. Simovici UMB, Doctoral Summer School Iasi, Romania What is Machine Learning? The Vapnik-Chervonenkis Dimension Probabilistic Learning Potential
More informationComputational Complexity and Intractability: An Introduction to the Theory of NP. Chapter 9
1 Computational Complexity and Intractability: An Introduction to the Theory of NP Chapter 9 2 Objectives Classify problems as tractable or intractable Define decision problems Define the class P Define
More informationA Lower Bound of 2 n Conditional Jumps for Boolean Satisfiability on A Random Access Machine
A Lower Bound of 2 n Conditional Jumps for Boolean Satisfiability on A Random Access Machine Samuel C. Hsieh Computer Science Department, Ball State University July 3, 2014 Abstract We establish a lower
More informationOutline. Complexity Theory. Introduction. What is abduction? Motivation. Reference VU , SS Logic-Based Abduction
Complexity Theory Complexity Theory Outline Complexity Theory VU 181.142, SS 2018 7. Logic-Based Abduction Reinhard Pichler Institut für Informationssysteme Arbeitsbereich DBAI Technische Universität Wien
More information1 The Probably Approximately Correct (PAC) Model
COS 511: Theoretical Machine Learning Lecturer: Rob Schapire Lecture #3 Scribe: Yuhui Luo February 11, 2008 1 The Probably Approximately Correct (PAC) Model A target concept class C is PAC-learnable by
More informationStatistical and Computational Learning Theory
Statistical and Computational Learning Theory Fundamental Question: Predict Error Rates Given: Find: The space H of hypotheses The number and distribution of the training examples S The complexity of the
More informationComputational Learning Theory for Artificial Neural Networks
Computational Learning Theory for Artificial Neural Networks Martin Anthony and Norman Biggs Department of Statistical and Mathematical Sciences, London School of Economics and Political Science, Houghton
More information1 Computational Problems
Stanford University CS254: Computational Complexity Handout 2 Luca Trevisan March 31, 2010 Last revised 4/29/2010 In this lecture we define NP, we state the P versus NP problem, we prove that its formulation
More informationP P P NP-Hard: L is NP-hard if for all L NP, L L. Thus, if we could solve L in polynomial. Cook's Theorem and Reductions
Summary of the previous lecture Recall that we mentioned the following topics: P: is the set of decision problems (or languages) that are solvable in polynomial time. NP: is the set of decision problems
More informationDecentralized Control of Discrete Event Systems with Bounded or Unbounded Delay Communication
Decentralized Control of Discrete Event Systems with Bounded or Unbounded Delay Communication Stavros Tripakis Abstract We introduce problems of decentralized control with communication, where we explicitly
More informationComputational Learning Theory
1 Computational Learning Theory 2 Computational learning theory Introduction Is it possible to identify classes of learning problems that are inherently easy or difficult? Can we characterize the number
More information20.1 2SAT. CS125 Lecture 20 Fall 2016
CS125 Lecture 20 Fall 2016 20.1 2SAT We show yet another possible way to solve the 2SAT problem. Recall that the input to 2SAT is a logical expression that is the conunction (AND) of a set of clauses,
More informationPropositional Resolution
Artificial Intelligence Propositional Resolution Marco Piastra Propositional Resolution 1] Deductive systems and automation Is problem decidible? A deductive system a la Hilbert (i.e. derivation using
More informationICML '97 and AAAI '97 Tutorials
A Short Course in Computational Learning Theory: ICML '97 and AAAI '97 Tutorials Michael Kearns AT&T Laboratories Outline Sample Complexity/Learning Curves: nite classes, Occam's VC dimension Razor, Best
More informationCS154, Lecture 15: Cook-Levin Theorem SAT, 3SAT
CS154, Lecture 15: Cook-Levin Theorem SAT, 3SAT Definition: A language B is NP-complete if: 1. B NP 2. Every A in NP is poly-time reducible to B That is, A P B When this is true, we say B is NP-hard On
More informationCMSC 858F: Algorithmic Lower Bounds Fall SAT and NP-Hardness
CMSC 858F: Algorithmic Lower Bounds Fall 2014 3-SAT and NP-Hardness Instructor: Mohammad T. Hajiaghayi Scribe: Philip Dasler September 23, 2014 The most important NP-Complete (logic) problem family! 1
More information