Hence C has VC-dimension d iff. π C (d) = 2 d. (4) C has VC-density l if there exists K R such that, for all

Transcription

1 1. Computational Learning Abstract. I discuss the basics of computational learning theory, including concept classes, VC-dimension, and VC-density. Next, I touch on the VC-Theorem, ɛ-nets, and the (p, q)-theorem for VC-classes. Definition 1.1. A concept class C on a set X is a subset of the powerset of X. Elements of C are known as concepts. Definition 1.2. Let C be a concept class on a set X: (1) for some Y X, C shatters Y if P(Y ) = C Y := {Y A : A C}. (2) C has VC-dimension d if there exists Y X with Y = d such that C shatters Y. (3) The shatter function of C is a function π C : ω ω given by π C (m) := max{ C Y : Y X, Y = m}. Hence C has VC-dimension d iff. π C (d) = 2 d. (4) C has VC-density l if there exists K R such that, for all m < ω, π C (m) Km l. (5) C is a VC-class if it has finite VC-dimension. (6) The dual concept class (on the set C) is given by C = {D C : ( x X)( A C)(x A A D)}. (7) The dual shatter function of C is π C = π C. (8) the VC-codimension of C is the VC-dimension of C. (9) the VC-codensity of C is the VC-density of C. Example 1.3. Let X = R and let C be the collection of open intervals in R. Any two-element subset of X is shattered but no three-element subset of X is shattered. So, the VC-dimension of C is 2. For any Y R with Y = m, there are ( ) m 2 elements of C Y with at least two elements, m elements with exactly one element, and 1 element with none. Hence, the shatter function is given by π C (m) = 1 2 m2 + 1 m + 1, for m 2. 2 Therefore, π C (m) = O(m 2 ), so C has VC-density 2. On can show that C has VC-codimension 2 and VC-codensity 1. Theorem 1.4 ([11]). A concept class C is a VC-class if and only if C is. 1

2 Theorem 1.5 (Sauer s Lemma). If C has VC-dimension n and m > n, then n ( ) m π C (m). i In particular, the VC-density is bounded above by the VC-dimension. Therefore, the following are equivalent for a concept class C: i=0 C is a VC-class, C has finite VC-dimension, C has finite VC-density, C has finite VC-codimension, C has finite VC-codensity. Definition 1.6. Let C be a concept class on a finite set X: (1) A subset T X is a transversal of C if, for all A C, A T. (2) The transversal number of C, denoted τ(c), is the minimal cardinality of a transversal of C. (3) A fractional transversal of C is a map σ : X [0, 1] such that, for each A C, x A σ(x) 1. (4) The fractional transversal number of C, denoted τ (C), is the infimum of x X σ(x) over all fractional transversals σ of C. (5) A subset D C is a packing of C if, for all A, B D, A B =. (6) The packing number of C, denoted ν(c), is the maximal cardinality of a packing of C. (7) A fractional packing of C is a map σ : C [0, 1] such that, for each x X, A C,A x σ(a) 1. (8) The fractional packing number of C, denoted ν (C), is the supremum of A C σ(a) over all fractional packings σ of C. Remark 1.7. For any concept class C on a finite set X, ν(c) τ(c), ν(c) ν (C), and τ (C) τ(c). Example 1.8. Let X = {0, 1, 2}, C = {{0, 1}, {1, 2}, {0, 2}}. Then we have that ν(c) = 1, τ(c) = 2, ν (C) = 3 2, and τ (C) = 3 2. Theorem 1.9. For every concept class C on a finite set X, τ (C) = ν (C). 2

3 Proof. This is proved via the Duality of Linear Programming. That is, if A is an m n real matrix, b R m, and c R n, if P = {x R n : x 0 and Ax b}, and D = {y R m : y 0 and y T A c T } are both non-empty, then min{c T x : x P } = max{y T b : y D}. Let A be the incidence matrix for X and C. So, A = (χ A (x)) A C,x X, where χ A : X {0, 1} is the characteristic function of A. Set b = 1 C and c = 1 X (tuples whose entries are 1). Check that τ (C) = min{c T x : x 0 and Ax b}, and ν (C) = max{y T b : y 0 and y T A c T }. The goal is now to find a condition to bound τ(c). Definition For ɛ > 0, µ a probability measure on X, and C a concept class of µ-measurable subsets of X, an ɛ-net of C is a transversal for the set C ɛ := {A C : µ(a) ɛ}. Theorem 1.11 (The VC-Theorem). Fix d, n < ω and ɛ > 0. Suppose that µ is a probability measure on X and C is a concept class on X (of µ-measurable sets) with VC-dimension d. Then, µ({a X n : {a 1,..., a n } is not an ɛ-net for C}) 2(2n) d 2 ɛn/2. Proof uses double counting method and Chebyshev s Inequality. Corollary Fix d < ω and r R. Then, there exists n R such that, for all finite concept classes C with VC-dimension d and ν (C) r, we have τ(c) n. Proof. Fix ɛ = 1/r and n large enough such that 2(2n) d 2 ɛn/2 < 1. As τ (C) = ν (C) r, 1/τ (C) ɛ. Let σ : X [0, 1] be an optimal fractional transversal and define a probability measure µ on X by letting x A µ(a) = 1 σ(x). For all A C, µ(a) ɛ by definition, hence τ (C) C = C ɛ. Then, the probability that a random n-element subset of X is not an ɛ-net (i.e., a transversal) for C is < 1. Hence, a n-element transversal exists. Definition A concept class C has the (p, q)-property if, for all D C with D = p, there exists a X such that {A D : a A} q. 3

4 In other words, every p-element subset of C has at least q elements that have a common intersection. Proposition 1.14 ([13]). For all 0 < q p and K < ω, there exists r R such that, for all finite concept classes C such that, C has the (p, q)-property and ( m q)(π C (m) < K( m q ) ), we have ν (C) r. The proof of this uses the Fractional Helly Property. Theorem 1.15 ((p, q)-theorem for VC-classes). If 0 < q p < ω and C is a VC-class on X with VC-codensity < q, then there exists n < ω such that, for all finite D C, if D has the (p, q)-property, then τ(d) n. (Moreover, n only depends on q, p, the VC-dimension of C, and the behavior of π C.) Proof. As C is a VC-class, choose d such that C has VC-dimension d and, since C has VC-codensity < q, choose K such that ( m q)(π C (m) < K( m q ) ). Let r be given by Proposition 1.14 for our q, p, and K. Then, let n be given by Corollary 1.12 for r and d. Proposition 1.14 gives us that ν (D) r and Corollary 1.12 gives us that τ(d) n. Suppose p = q. Then the (q, q)-theorem is, in effect, a finite version of compactness for NIP Theories. It says that, for a finite VC-class C, if every q element subset has a common intersection, then there is a uniform bound on the number of elements needed to cover C. 4

5 2. Introduction to NIP Theories Abstract. First, I conclude computational learning by discussing two notions of learning. Then, I discuss the notion of UDTFS and how it relates to sequence compression schemes. Finally, I begin talking about NIP Theories. Let X be any set and C a concept class on X. Let P fin (X) be the finite subsets of X. Definition 2.1. We say that C is probably approximately correctly learnable (PAC-learnable) if there exists H Y : Y P fin (X) a sequence of hypothesis functions H Y : C Y P(X) such that, for all ɛ > 0 and all δ > 0, there exists N ɛ,δ < ω such that, for all n N ɛ,δ, all A C, and all probability measures µ on X, the µ-probability that a randomly chosen a 1,..., a n X satisfies is < δ. µ(a H {a1,...,a n}(a {a 1,..., a n })) > ɛ Theorem 2.2 (PAC-Learning Theorem, [3]). For any concept class C, the following are equivalent: C is a VC-class, C is PAC-learnable. This is proved using the VC-Theorem from last talk. Example 2.3 (Example 1.3, Revisited). Consider X = R and C the concept class of all open intervals. For every finite Y R and every A Y, let H Y (A) = {x R : ( a, b A)(a x b)}. Then, one can show that this witnesses that C is PAC-learnable with N ɛ,δ 2 ( ) 2 ɛ log. δ The function ɛ, δ N ɛ,δ is called the sample complexity of the hypothesis function H = H Y : Y P fin (X). Definition 2.4. We say that C has a d-dimensional sequence compression scheme if there exists a sequence of compression functions κ Y : Y P fin (X) (with κ Y : C Y Y d ) and a finite number of recovery functions ρ i : i < K (with ρ i : X d P(X)) such that, for each finite non-empty Y X and A C Y, there exists i < K such that ρ i (κ Y (A)) Y = A. 5

6 Example 2.5 (Example 1.3, Revisited). Consider X = R and C the concept class of all open intervals. Consider the compression κ Y (A) = min A, max A for A and κ Y ( ) = min Y, max Y. Consider the recovery functions ρ 1 (a, b) = {x R : a x b} and ρ 2 (a, b) =. Check that, for any non-empty finite Y R and any A C Y, ρ 1 (κ Y (A)) Y = A if A, and ρ 2 (κ Y ( )) Y =. There is no 1-dimensional compression scheme (by the following proposition). Proposition 2.6. If C has a d-dimensional sequence compression scheme, then C has VC-density d. In particular, C is a VC-class. Proof. For any finite non-empty Y X, an element A C Y is determined by an element in Y d (namely, κ Y (A)) and ρ i for some i < K. Hence, C Y K Y d. Conjecture 2.7 (Warmuth Conjecture). If C is a VC-class, then C has a sequence compression scheme. Proposition 2.8 ([7]). The following hold for a concept class C: If C has VC-density < 2, then C has a sequence compression scheme. If C has VC-dimension 1, then C has a 1-dimensional sequence compression scheme. Let T be a complete, first-order theory in a language L with monster model U. We will bring all the definitions and results from the previous talk into model theory with the following convention: Definition 2.9. Fix a partitioned formula ϕ(x; y). For a U x and B U y, let ϕ(a; B) := {b B : U = ϕ(a; b)}. Now consider the following concept class on U y, C U ϕ := {ϕ(a; U y ) : a U x }. From this, we get: Say ϕ has VC-dimension n if C U ϕ does. Say ϕ has VC-density l if C U ϕ does. Say ϕ has NIP if C U ϕ is a VC-class. Say T has NIP if every partitioned formula ϕ(x; y) does. 6

7 Example The following theories have NIP: T = Th(C; +,, 0, 1), the theory of algebraically closed fields (of characteristic 0). T = Th(R; +,, <, 0, 1), the theory of real closed fields. T = Th(Q p ; +,, 0, 1), the theory of the p-adic field. T = Th(Q; +, <), the theory of densely ordered abelian groups. The theory of Rado s random graph does not have NIP. Definition A formula ϕ(x; y) has uniform definability of types over finite sets (UDTFS) (of rank d) if there exists finitely many formulas ψ i (y; z 1,..., z d ) : i < K such that, for all finite B U y and all a U x, there exists c 1,..., c d B and i < K such that ϕ(a; B) = ψ i (B; c 1,..., c n ). Proposition 2.12 ([1]). If T is weakly o-minimal, then ϕ(x; y) had UDTFS-rank x. In particular, this holds for real closed fields and densely ordered abelian groups, as these theories are weakly o-minimal. Theorem 2.13 ([10]). If ϕ has UDTFS-rank d, then C U ϕ has a d- dimensional sequence compression scheme. Proof. Let ψ i (y; z 1,..., z d ) : i < K witness that ϕ(x; y) has UDTFSrank d. For each i < K, consider the recovery function ρ i : U d y P(U y ) given by ρ i (c 1,..., c d ) = ψ i (U y ; c 1,..., c d ). For any finite non-empty B U y, consider the compression function κ B : C U ϕ B B d given by: For a U x, let c 1,..., c d B and i < K be such that ϕ(a; B) = ψ i (B; c 1,..., c d ). Then, set Hence, κ B (ϕ(a; B)) = c 1,..., c d. ρ i (κ B (ϕ(a; B))) B = ψ i (B; c 1,..., c d ) = ϕ(a; B). Therefore, this is a d-dimensional sequence compression scheme for Cϕ U. Example Consider ϕ(x 1, x 2, x 3 ; y 1, y 2 ) := (x 1 y 1 ) 2 + (x 2 y 2 ) 2 x 2 3 in real closed fields. Then, C R ϕ is the set of all closed disks in R 2. By Proposition 2.12 and Theorem 2.13, it has a 3-dimensional sequence compression scheme. Corollary If ϕ has UDTFS-rank d, then ϕ has VC-density d. In particular, if ϕ has UDTFS, then it has NIP. 7

8 What about the converse? Conjecture 2.16 (UDTFS Conjecture, [7]). If ϕ(x; y) has NIP, then ϕ has UDTFS. By Theorem 2.13, the UDTFS Conjecture implies the Warmuth Conjecture. Proposition 2.17 ([7]). Fix a partitioned formula, ϕ(x; y). (1) If ϕ has VC-density < 2, then ϕ has UDTFS. (2) If ϕ had VC-dimension 1, then ϕ has UDTFS-rank 1. In particular, if C is a concept class with VC-density < 2, then C has a sequence compression scheme and, if C is a concept class with VC-dimension 1, then C has a 1-dimensional sequence compression scheme. So this proves Proposition 2.8. Proof of Proposition 2.17 (2). Without loss of generality, we may suppose that, for all b, c U y, at least one of the following holds: = ( x)(ϕ(x; b) ϕ(x; c)), = ( x)(ϕ(x; c) ϕ(x; b)), or = ( x)(ϕ(x; b) ϕ(x; c)). This puts a definable quasi-forest order ϕ on U y, namely b ϕ c if = ( x)(ϕ(x; b) ϕ(x; c)). Let ψ(y; z) = [z ϕ y]. Then, for any a U x and finite B U y, choose c B such that = ϕ(a; c) and c is ϕ -minimal such in B. Then, it is easy to check that ϕ(a; B) = ψ(b; c). Theorem 2.18 ([7]). If T is dp-minimal, then all formulas have UDTFS. Example The p-adics are dp-minimal. Hence, the concept class of all closed balls in the p-adic field have a sequence compression scheme. Use ϕ(x 1, x 2 ; y) := v p (y x 1 ) v p (x 2 ). The following partial result to the UDTFS Conjecture is due to Artim Chernikov and Pierre Simon: Theorem 2.20 ([5]). If T has NIP, then all formulas have UDTFS. We will spend the final talk proving this theorem. 8

9 3. The UDTFS Conjecture Abstract. Starting with honest definitions, I discuss Chernikov and Simon s partial solution to the UDTFS Conjecture and the ramifications this has on computational learning theory. Let T be a complete, first-order theory in a language L with monster model U. Recall the definition of UDTFS: A formula ϕ(x; y) has UDTFS if there exists finitely many formulas ψ i (y; z) : i < K such that, for all finite B U y and all a U x, there exists c B n and i < K such that ϕ(a; B) = ψ i (B; c). Recall also Theorem 2.20 by Chernikov and Simon: If T has NIP, then all formulas have UDTFS. We prove this now. Definition 3.1. Fix C U a set, I, < a linear order, and b i U for each i I (all of the same sort). We say that b := b i : i I is an indiscernible sequence over C if, for all n < ω, for all i 1 <... < i n and j 1 <... < j n from I, for all L(C)-formulas δ(y 1,..., y n ), = δ(b i1,..., b in ) δ(b j1,..., b jn ). Proposition 3.2. A formula ϕ(x; y) has NIP if and only if there exists n < ω such that, for all indiscernible sequences b = b i : i I and all a U x, there do not exist i 0 <... < i n such that for all l < n. = ϕ(a; b il ) ϕ(a; b il+1 ) Proof. ( ): If ϕ(x; y) does not have NIP, then by compactness and Ramsey s Theorem, there exists an indiscernible sequence b i : i < ω such that, for any I ω, {ϕ(x; b i ) : i I} { ϕ(x; b i ) : i ω \ I} is consistent. In particular, for I the even numbers, we have infinite alternation. ( ): Fix n and suppose a and b i : i < 2n are such that = ϕ(a; b i ) if and only if i is even. For any s : n 2, consider the formula ( ) x ϕ(x; b 2i+s(i) ) s(i). i<n As this is true, by indiscernibility, the following is also true ( ) x ϕ(x; b i ) s(i). i<n As s n 2 was arbitrary, ϕ has VC-dimension n. 9

10 Definition 3.3. Fix a global type p(x) S x (U) and a set A U. p is invariant over A if, for all σ Aut(U/A), for all ϕ(x; y), and all b U y, ϕ(x; b) p(x) if and only if ϕ(x; σ(b)) p(x). p is definable over A if, for all formulas ϕ(x; y), there exists an L(A)-formula ψ(y) such that, for all b U y, ϕ(x; b) p(x) if and only if = ψ(b). p is finitely satisfiable over A if, for all formulas ϕ(x; b) in p(x), there exists a A such that = ϕ(a; b). Lemma 3.4. If p(x) S x (U) is definable over A or finitely satisfiable over A, then p is invariant over A. Proof. (1): If p(x) is definable over A, fix ϕ(x; y) and suppose ψ(y) is an L(A)-formula defining ϕ(x; y) in p. Then, for any b U y and σ Aut(U/A), ϕ(x; b) p(x) iff. = ψ(b) iff. = ψ(σ(b)) iff. ϕ(x; σ(b)) p(x). (2): Suppose p(x) is finitely satisfiable over A. If p(x) is not invariant over A, then fix ϕ(x; y), b U, and σ Aut(U/A) witnessing this. That is, ϕ(x; b) p(x) but ϕ(x; σ(b)) p(x). Hence, there exists a A such that = ϕ(a; b) ϕ(a; σ(b)). As ϕ(a; y) is over A, this is a contradiction. Definition 3.5. For a set A U, define Sx inv (U, A) := {p S x (U) : p is invariant over A}, S fin x (U, A) := {p S x (U) : p is finitely satisfiable over A}, Sx def (U, A) := {p S x (U) : p is definable over A}. Lemma 3.6. For all A, each of Sx inv (U, A), Sx fin (U, A), and Sx def (U, A) are closed subsets of the compact space S x (U), hence each are compact. Proposition 3.7. If p(x) S x (U) is invariant over A and a i : i < ω is such that, for all i < ω, a i = p A {aj :j<i}, then a i : i < ω is an indiscernible sequence over A. We call such sequences Morley sequences of p over A. Fix M = T and B M y for some variable y. Define a new language L P := L {P (y)}. M, B is the obvious L P -structure. Fix ϕ(x; y) a formula and a M x. As U is a monster model for T, all models (even L P -structures) will have universes contained in U. Definition 3.8. We say that an L-formula ψ(y; z) is an honest definition of the ϕ-type tp ϕ (a/b) if there exists an elementary extension 10

11 M, B M, B and c (B ) n such that ϕ(a; B) ψ(b ; c) ϕ(a; B ). In particular, ϕ(a; B) = ψ(b; c), so ψ(y; c) defines tp ϕ (a/b). Theorem 3.9 ([4]). If ϕ(x; y) has NIP, a M x, B M y, then tp ϕ (a/b) has an honest definition. To prove this, we choose a type q in the space Sy fin (U, B) and build a Morley sequence of q over B in B enforcing alternation of ϕ(a; y). By NIP, the alternation stops and, by compactness, we get θ q (y) q(y) and t q < 2 such that θ q (y) P (y) ϕ(a; y) tq. We use the compactness of Sy fin (U, B) to conclude. Proposition We have ψ(y; z) is an honest definition of tp ϕ (a/b) if and only if, for all finite B 0 ϕ(a; B), there is c B n such that B 0 ψ(b; c) ϕ(a; B). This is proved using the compactness of S y (B). So, for any NIP formula ϕ(x; y), any B U y, and any a U x, there exists ψ(y; z) such that, for any finite B 0 ϕ(a; B), there exists c B n so that B 0 ψ(b; c) ϕ(a; B). In particular, if B is finite and B 0 = ϕ(a; B), we get ϕ(a; B) = ψ(b; c). So we only need uniformity of ψ (i.e., independence from a and B). By compactness, we get the following lemma. Lemma Let ϕ(x; y) be an L-formula. For any function η from L-formulas to ω, there are finitely many formulas ψ j (y; z j ) (for j < m) such that, for all M = T, B M y, and a M x, there exists j < m such that, for all B 0 ϕ(a; B) with B 0 η(ψ j ), there exists c B n such that B 0 ψ j (B; c) ϕ(a; B). Lemma Let ϕ(x; y) be an L-formula. There exists finitely many formulas ψ 0 (y; z 0 ),..., ψ m 1 (y; z m 1 ) such that, for all M = T, B M y, and a M x, for all B 0 ϕ(a; B) finite, there exists c B n and j < m such that B 0 ψ j (B; c) ϕ(a; B). 11

12 Proof of Lemma For each ψ(y; z), let η(ψ) be the VC-density of ψ. Then there exists θ 0 (y; z),..., θ m 1 (y; z) satisfying the conclusion of the previous lemma for η. Let p = q = η(θ j ) and we get K j by the (p, q)-theorem. Let K = max{k j : j < m} and, for each j < m, define ψ j (y; z 0,..., z K 1 ) := θ j (y; z l ). This works. Fix M = T, B M y, a M x, B 0 ϕ(a; B) finite. For j < m in the previous lemma, define Define a concept class on C, l<k C := {c B n : θ j (B; c) ϕ(a; B)}. D := {θ j (b; C) : b B 0 }. This satisfies the (p, q)-property (with p = q = η(θ j )) by the previous lemma. [This is because, for each B 1 B 0 (the index set for D) with B 1 = q = η(θ j ), there exists c B n so that B 1 θ j (B; c) ϕ(a; B), hence c C and c b B 1 θ j (b; C).] By the (p, q)-theorem (and our choice of K j K), there exists a transversal C 0 C with C K. If c 0 enumerates C 0, then B 0 ψ j (B, c 0 ) ϕ(a; B). Setting B 0 = ϕ(a; B) for finite B, this concludes the proof of Theorem

13 References [1] M. Aschenbrenner, A. Dolich, D. Haskell, D. MacPherson, and S. Starchenko, Vapnik-Chervonenkis density in some theories without the independence property, I, Preprint (2011). [2], Vapnik-Chervonenkis density in some theories without the independence property, II, Preprint (2011). [3] A. Blumer, A. Ehrenfeucht, D. Haussler, and M. Warmuth, Learnability and Vapnik-Chervonenkis Dimension, Journal of the Association for Computing Machinery 36 (1989), no. 4, [4] A. Chernikov and P. Simon, Externally definable sets and dependent pairs, Israel Journal of Mathematics (2012). to appear. [5], Externally definable sets and dependent pairs, II (2012). preprint. [6] S. Floyd and M. Warmuth, Sample compression, learnability, and the Vapnik- Chervonenkis dimension, Machine Learning (1995), [7] V. Guingona, On uniform definabiliy of types over finite sets, J. Symbolic Logic 77 (2012), no. 2, [8] H. R. Johnson, dp-rank and forbidden configurations, Notre Dame Journal of Formal Logic 54 (2013), no. 1. [9], Vapnik-Chervonenkis density on indiscernible sequences, stability, and the maximum property (2013). preprint. [10] H. R. Johnson and M. C. Laskowski, Compression schemes, stable definable families, and o-minimal structures, Discrete Comput. Geom. 43 (2010), no. 4, [11] M. C. Laskowski, Vapnik-Chervonenkis classes of definable sets, J. London Math. Soc. 245 (1992), [12] J. Matousek, Lectures on Discrete Geometry, Springer-Verlag New York Inc., [13], Bounded VC-dimension implies a fractional Helly theorem, Discrete Comput. Geom. 31 (2004), [14] S. Shelah, Dependent first order theories, continued, Israel Journal of Mathematics 173 (2009), [15], Classification Theory and the Number of Non-Isomorphic Models. [16] P. Simon, On dp-minimal ordered structures, J. Symbolic Logic 76 (2011), [17], Invariant types in dp-minimal theories (2012). preprint. [18], Lecture notes on NIP theories, preprint. Ben-Gurion University of the Negev URL: address: guingona@math.bgu.ac.il 13