Vitaly Bergelson and Tomasz Downarowicz May 22, Introduction Let (X, B, µ, T ) be an invertible ergodic probability measure preserving system.

LARGE SETS OF INTEGERS AND HIERARCHY OF MIXING PROPERTIES OF MEASURE-PRESERVING SYSTEMS Vitaly Bergelson and Tomasz Downarowicz May 22, 2007 Abstract. We consider a hierarchy of notions of largeness for subsets of Z (such as thick sets, syndetic sets, IP-sets, etc., as well as some new classes) and study them in conjunction with recurrence in topological dynamics and ergodic theory. We use topological dynamics and topological algebra in βz to establish the connections between various notions of largeness and apply the obtained results to the study of sets of fat intersections RA,B ɛ = {n Z : µ(a T n B) > µ(a)µ(b) ɛ}. Among other things we show that the sets RA,B ɛ allow one to distinguish between various notions of mixing and introduce an interesting class of weakly but not mildly mixing systems. Some of our results on fat intersections are established in a more general context of unitary Z-actions. Introduction Let (X, B, µ, T ) be an invertible ergodic probability measure preserving system. Given ɛ > 0 and A, B B with µ(a) > 0, µ(b) > 0, let us define the set of fat intersections by R ɛ A,B = {n Z : µ(a T n B) > µ(a)µ(b) ɛ}. When A = B, the sets RA,B ɛ are intrinsically connected with the various enhancements and applications of the classical Poincaré recurrence theorem and are relatively well understood. For example, the Khintchine recurrence theorem ([Kh]; see also [B1], Section 5) says that for any, not necessarily ergodic, probability measure preserving system (X, B, µ, T ), any A with µ(a) > 0 and any ɛ > 0, the set RA,A ɛ 2000 Mathematics Subject Classification. 37A25, 37B20. Key words and phrases. Weak mixing, mild mixing, fat intersections, IP-sets, idempotents, central sets, upper Banach density. The first author is partially supported by NSF grant DMS-0600042. Research of the second author is supported from resources for science in years 2005-2008 as research project (grant MENII 1 P03A 021 29, Poland) 1 Typeset by AMS-TEX

2 VITALY BERGELSON AND TOMASZ DOWNAROWICZ is syndetic (i.e., has bounded gaps). This result, in turn, follows from (a stronger) fact that RA,A ɛ is a -set, namely a set which has nontrivial intersections with any set of the form {n i n j } i>j, where (n i ) i N is an injective sequence in Z (see Theorem 3.1 below). Note that while every -set is syndetic, not every syndetic set is a -set (consider for example the set of all odd numbers). Assuming ergodicity, one can show that the sets RA,B ɛ are always syndetic. On the other hand, the natural question whether the sets RA,B ɛ are always of the form E + k, where E is a -set, k Z, has, in general, a negative answer (see Theorem 1.7 below). One of the goals of this paper to introduce and study some new notions of largeness with the intention to better understand the sets of fat intersections and to apply them to the study of mixing properties of dynamical systems. In order to formulate our main results we have first to introduce and discuss the pertinent notions of largeness. This is done in Section 1, at the end of which the formulations of our main theorems are given. In Section 2 we take a closer look at notions of largeness which are intrinsically related to topological dynamics. In particular, we show that one of the notions playing the decisive role in this paper, namely that of D-sets (see the definition in Section 1), can be naturally viewed as the extension of Furstenberg s notion of central sets (see [F], p. 161) which proved to be very useful in various applications of Ergodic Theory to combinatorics. (See for example [B1] and [B-M]). In Section 3 we provide the proofs of the characterizations of ergodicity, weak, mild and strong mixing in terms of sets of fat intersections. In Section 4 we give an example of a dynamical system which not only proves that two of the classes under study (IP + and D ) are not contained in one another, but also that one cannot replace D by its intersection with IP + in the characterization of the weak mixing property. Finally, in Section 5 we apply our notions to isolate certain nonempty subclass of weakly mixing but not mildly mixing transformations. The paper is concluded by an Appendix containing an explicit example of a topological dynamical system with specific properties. Besides being interesting in its own right, the existence of such a system is important in one of the proofs in Section 2. Acknowledgement. We are greatful to Sarah Bailey-Frick, Ronnie Pavlov and Neil Hindman for useful comments. We also thank the anonymous referee for numerous pertinent remarks and suggestions. Section 1. Notions of largeness via duality Let F be a family of nonempty subsets of the integers Z. By F we will denote the dual family consisting of all sets G such that G F for every F F. The family F is partition regular if, whenever F F is represented as a union of finitely many sets, then at least one of them belongs to F. It is not hard to verify that if F is partition regular then its dual F is a filter: the intersection of two elements of

LARGE SETS OF INTEGERS 3 F belongs to F. (The other requirement for a filter, the property of being closed under taking supersets, is obvious for F.) Two elementary examples of this kind are as follows: 1. Fix some n 0 Z and let F = {F Z : n 0 F }. Then F = F. 2. Let F = I = {F Z : F = } (infinite sets). Then F = I = {F Z : Z \ F < } (cofinite sets). Let us now mention a more subtle example. 3. A set F Z is called an IP-set if it contains the set FS(S) of finite sums of some sequence of nonzero integers S = (s n ) n 1 : FS(S) = {s n1 + s n2 + + s nk : n 1 < n 2 < < n k, k N}. Let IP be the family of all IP-sets. One can show that both IP-sets and IP - sets (members of the dual family IP ) can be characterized (with the help of Hindman s theorem) in terms of idempotents in βz (see Definition 1.2 below and Theorems 1.2 and 1.5 in [B2]). Recall that a family of subsets of Z which is both partition regular and a filter is called an ultrafilter (or a maximal filter). Note the obvious fact that the union of any collection of ultrafilters is partition regular. Also, while an intersection of ultrafilters need not be an ultrafilter, it is always a filter. The collection of all ultrafilters is denoted by βz and, endowed with an appropriate topology, becomes the Stone-Čech compactification of Z. There is a natural semigroup structure in βz extending the addition operation of Z (for more details see [H-S]). The above examples have interpretation in terms of ultrafilters, as follows: In the first one, F is nothing but a so-called principal ultrafilter, i.e., the ultrafilter representing n 0 in βz (and so is F ). In the second, F is the union of all not principal ultrafilters and F is the intersection of all such ultrafilters. Finally, in the third example F is the union of all not principal ultrafilters which are idempotents for the natural semigroup structure of βz (that is, F is the union of all idempotents except zero) and F is the intersection of the nonzero idempotents (cf. [B2, Theorem 2.15 (i), p.20]). The above facts are special cases of the following more general statement: Lemma 1.1. (1) If F is an ultrafilter then F = F. (2) If F = α F α then F = α F α. In particular, whenever F is a union of some collection of ultrafilters, then F is the intersection of the same collection of ultrafilters. Intuitively, if we have a union of a rich collection of families, its dual contains relatively few very large sets, namely, sets which intersect nontrivially every member of every family in this collection. This approach to largeness will be utilized throughout this paper: a set is large if it belongs to the dual of a rich family of

4 VITALY BERGELSON AND TOMASZ DOWNAROWICZ sets containing a union of many ultrafilters. For this reason the first example above is not very useful: the family F is just a single ultrafilter (so is F ), moreover, F contains finite sets, so being a member of F cannot be considered a criterion for largeness. But leaving this exceptional example aside, we will investigate a whole hierarchy of notions of largeness constructed with the help of dual families, of which the property of being a member of I is the strongest. Several important notions of largeness can be introduced with the help of idempotent ultrafilters. In order to facilitate the discussion we list some of the important families of large sets in the following definition. (Note that the family IP appearing in item (1) below was already introduced above.) Definition 1.2. (1) The collection IP (of IP-sets) is the union of all nonzero idempotents 0 p βz. Accordingly, IP is the intersection of all nonzero idempotents. (2) The collection D (of D-sets) is the union of all idempotents p βz such that every member of p has positive upper Banach density 1. Accordingly, D is the intersection of all such idempotents. (3) The collection C (of C-sets or central sets) is the union of all minimal idempotents. 2 Accordingly, C is the intersection of all minimal idempotents. Since every member of a minimal idempotent has positive upper Banach density 3, we have C D, hence, directly from the definitions, we obtain the following hierarchy: I IP D C C D IP I. As we will see below all these inclusions are in fact proper. We introduce two more notions of largeness defined via duality, as follows: Definition 1.3. (1) A subset F Z is called a -set or we say that F belongs to the family, if there exists an injective sequence of integers S = (s n ) n 1 such that the difference set (S) = {s i s j : i > j} is contained in F. (2) A set F Z is thick if it contains arbitrarily long intervals [a, b] = {a, a + 1, a + 2,..., b}. The collection of all thick sets will be denoted by T. The dual family T is easily seen to coincide with the collection of all syndetic sets (i.e., sets having bounded gaps). 1 1 Upper Banach density of a set E Z is defined as lim sup (m n) E [n, m 1]. If m n the corresponding limit exists then it is called the Banach density of E. 2 An idempotent is minimal if it belongs to a minimal right ideal in βz, see [H-S] and [B2] for details. Also see the discussion in Section 2 on various equivalent definitions of the notion of central set. 3 This follows from a stronger fact that every member of a minimal idempotent is piecewise syndetic, see [B2], Theorem 2.4 and Exercise 7.

LARGE SETS OF INTEGERS 5 The family is a union of a collection of ultrafilters (see [B-H2, Definition 1.6 and Lemma 1.9]), while that of thick sets is not (because it is not partition regular). It is known (and not very hard to see) that every thick set is an IP-set and that every IP-set is a -set, but not the other way around. In particular, the collection of ultrafilters whose union is contains more than just idempotents. The hierarchy of notions of largeness introduced so far is as follows: cofinite = I IP D C T = syndetic Given a family F and k Z, the shifted family is defined by F +k = {F +k : F F}, where F + k = {n + k : n F }. The extreme classes in the above diagram are shift invariant; a shifted cofinite set remains cofinite, a shifted syndetic set remains syndetic. The other classes fail to be shift invariant. This is not surprising for notions involving idempotents due to the simple fact that if p is an idempotent then p + k is not (unless k = 0). To see that the family is not shift invariant note that it contains the set of all even integers while it does not contain the set of all odd integers. When F is not shift-invariant, there are two natural ways of building a shift invariant family from it: Definition 1.4. For a given family F, F + denotes the union k Z (F + k) while F denotes the intersection k Z (F + k). When applying these operations to a dual family F, we will write F+ and F, skipping the parentheses in what should formally be (F ) + and (F ). Such convention complies with the existing notation e.g. for IP +-sets. We will call F+ the extended dual family. Note that in general, F+ is not a dual family. On the other hand, by Lemma 1.1 (2), the family F is the dual of F + (it could be written as (F + ), but we will not use this confusing symbol). The elements of F are essentially larger than the members of F as they must intersect every set in the family F + which is much richer than F. If F is a union of ultrafilters, so is F +, thus F is an intersection of ultrafilters, and in particular is a filter. It seems that the type F of shift invariant families has not been sufficiently recognized in the existing literature. Here is the diagram including all dual and extended dual classes related to the families discussed so far: I IP D C T I IP D C T I+ + IP + D+ C+ T+ We will show now that in this diagram no inclusions hold except the ones that are shown and those obtained by composition. First, observe the following property

6 VITALY BERGELSON AND TOMASZ DOWNAROWICZ of all -sets F : certain distance between elements of F appears infinitely many times. Indeed, in any difference set (S), with S = (s n ) the distance s 2 s 1 occurs between all pairs of elements s n s 1 and s n s 2 (n > 2). Obviously, the same property holds for shifted -sets. We conclude that the set of powers of 2 does not contain any shift of any -set, which implies that the complement of powers of 2 is. Hence the family is larger than the class of cofinite sets I. Further, the set of all odd numbers is a +-set and is not an IP-set, hence in the diagram it escapes any class contained in C. Likewise, the set of all even integers is a -set and not C. The construction of an IP but not + is provided in Theorem 2.11 (1). The existence of a D not IP +-set will follow from Theorem 1.7 below. A C but not D + example is our Theorem 2.11 (2). Finally, a syndetic set which is not C + is provided in [B2], Theorem 2.10. All other unwanted inclusions are now eliminated by superposition. It is worth noticing that the family C + (shifted central sets) coincides with PS, the family of piecewise syndetic or PS-sets (a set is piecewise syndetic if it is an intersection of a thick set and a syndetic set). The proof can be found in [H-S, Theorem 4.43 (c)]. Thus, C = PS, the dual to the family of piecewise syndetic sets. Elements of this dual can be easily identified as syndetically thick, meaning that for every E PS and n 1 intervals of length n appear in E with bounded gaps (such sets have been introduced in [D] as S-sets). This paper focuses on the role the notions of largeness of subsets of Z play in ergodic theory and topological dynamics. Recall that (X, T ) is a (topological) dynamical system if X is a compact Hausdorff space and T : X X is a homeomorphism. The families defined as unions of certain idempotents (IP-sets, C-sets and D-sets) have interpretations (and indeed convenient alternative definitions) as families of sets of the form {n Z : (T n x, T n y) U}, where y is a recurrent point, the pair (x, y) is proximal 4 and U is a neighborhood of (y, y) in X X. While the families of IP-, C- and D-sets are useful in topological dynamics, their dual and extended dual families find applications in ergodic theory. For example we will show how notions of largeness such as D +, D and IP can be used to characterize the familiar ergodic-theoretic notions of ergodicity, weak mixing and mild mixing. As it was already mentioned in the Introduction, in this paper we study the sets of fat intersections R ɛ A,B = {n Z : µ(a T n B) > µ(a)µ(b) ɛ}. In the spirit of Khintchine s Theorem we will locate the sets of fat intersections for specific types of systems in our diagram of large sets. First of all, the Khintchine 4 Two points x, y in a topological dynamical system (X, T ) are proximal if the set of pairs (T n x, T n y) has an accumulation point on the diagonal.

LARGE SETS OF INTEGERS 7 theorem can be strengthened: the set RA,A ɛ is always (see Theorem 3.1). It is not very surprising that the sets RA,A ɛ do not form a shift invariant family. However, to capture the fat intersections for two arbitrary sets A and B (this only makes sense in ergodic systems) one needs a shift invariant notion simply because R ɛ A,T k B = Rɛ A,B + k. The most natural candidate, namely the class +, turns out to be too restrictive. The sets of fat intersections are in this class only for certain rather special types of systems, e.g. systems with discrete spectrum. The smallest class in our diagram that suffices for all ergodic systems is the extended dual D+. However, curiously enough, we will show that for the notions of mixing under study, the sets RA,B ɛ are captured by the more restrictive shift invariant dual of the form F : for weak mixing this is D, for mild mixing this is IP, and for mixing, directly by the definition, this is I (which can be also written as I ). Let us briefly recall some of the ergodic-theoretic notions: Definition 1.5. (1) The system (X, B, µ, T ) has discrete spectrum if L 2 (µ) is spanned by the eigenfunctions of the unitary operator induced by T. (2) The system (X, B, µ, T ) is weakly mixing if the product system (X X, B B, µ µ, T T ) is ergodic. (3) The system (X, B, µ, T ) is mildly mixing if there are no nontrivial rigid L 2 - functions. (A function f L 2 (µ) is rigid if T n k f f in L 2 for some sequence n k.) (4) The system (X, B, µ, T ) is mixing if for any two sets A, B B one has µ(a T n B) µ(a)µ(b) as n. We stress that the appropriate categorization of fat intersections for all pairs of sets is in many cases equivalent to a given ergodic-theoretic notion, which makes the hierarchy of largeness very useful. In the following theorem we collect formulations of various familiar notions of mixing in terms of sets RA,B ɛ (see also Final remarks at the end of the paper). Some of the items in Theorem 1.6 below are mere reformulations of well known facts - see for example [F], others have relatively easy proofs provided in Section 3 (see also Remark 1 below). Given a system (X, B, µ, T ) we denote by R(X, B, µ, T ) the family of all sets of fat intersections in this system, R(X, B, µ, T ) = {RA,B ɛ : ɛ > 0, A, B B}. Theorem 1.6. Let (X, B, µ, T ) be an invertible probability measure preserving system. Then: (1) For any A B and any ɛ > 0 we have R ɛ A,A. (2) If (X, B, µ, T ) is ergodic and has discrete spectrum then R(X, B, µ, T ) +. (3) (X, B, µ, T ) is ergodic R(X, B, µ, T ) D+ R(X, B, µ, T ) C+ R(X, B, µ, T ) T. (4) (X, B, µ, T ) is weakly mixing R(X, B, µ, T ) D R(X, B, µ, T ) D R(X, B, µ, T ) C R(X, B, µ, T ) C.

8 VITALY BERGELSON AND TOMASZ DOWNAROWICZ (5) (X, B, µ, T ) is mildly mixing R(X, B, µ, T ) IP R(X, B, µ, T ) IP (c.f. Chapter 9, Section 4 in [F]). (6) (X, B, µ, T ) is mixing R(X, B, µ, T ) I R(X, B, µ, T ) R(X, B, µ, T ) (see [K-Y] and Remark 1 (f) below). Remark 1. Some of the equivalences in Theorem 1.6 are trivial or very easy, some others follow from known results: (a) It is clear that in (3) only the first equivalence needs a proof, the other two follow from inclusions of the families of sets and from the fact that in nonergodic systems the family R(X, B, µ, T ) contains the empty set, so R(X, B, µ, T ) T. (b) Since for any system (X, B, µ, T ) the family R(X, B, µ, T ) is shift invariant, it is obvious that R(X, B, µ, T ) F R(X, B, µ, T ) F for any family F. (c) Notice that if R(X, B, µ, T ) F, where F is a filter, then, intersecting each set RA,B ɛ with the corresponding set Rɛ A,Bc, we obtain that the sets of accurate intersections Q ɛ A,B = {n Z : µ(a T n B) µ(a)µ(b) < ɛ} also belong to F. In other words, Q(X, B, µ, T ) = {Q ɛ A,B : ɛ > 0, A, B B} F. (Clearly, since Q ɛ A,B Rɛ A,B, the converse implication also holds.) Thus the statements (4),(5) and (6) in Theorem 1.6 are equivalent to analogous statements with R(X, B, µ, T ) replaced by Q(X, B, µ, T ). (d) If the system (X, B, µ, T ) is not weakly mixing then one can find two sets A and B and an ɛ > 0 such that the sets RA,A ɛ and Rɛ A,B are disjoint (c.f. Theorem 4.31 in [F]), and so they cannot both be C -sets. Thus the condition R(X, B, µ, T ) C implies weak mixing. 5 Hence, using the remark (b) and obvious inclusions, we conclude that also in (4) only the first equivalence needs a proof. In fact, the first implication = can be deduced (using (c)) from the classical fact that weak mixing is equivalent to the condition for any sets A, B B. n 1 1 lim µ(a T i B) µ(a)µ(b) = 0 n n i=0 (e) The first equivalence in (5) (in terms of accurate intersections) is Proposition 9.22 [F], the second follows from (b). (f) The first equivalence in (6) applied to accurate intersections becomes merely the definition of mixing. The second equivalence in (6) (formulated for accurate 5 The same fact is proved (by a different method) in [K-Y] Proposition 5.2, in response to the question formulated in the preliminary version of this paper.

LARGE SETS OF INTEGERS 9 intersections) is nontrivial and has been recently proved in [K-Y] Theorem 4.4 (see also [K-Y] Proposition 5.1, formulated in response to the question in the preliminary version of our paper). To summarize the content of the above remark, only (1), (2) and portions of (3) and (4) require proofs (see Theorems 3.1, 3.2, 3.8 and 3.9 in the next section, respectively). For completeness we will also provide a proof of (5) using the language of idempotents (see Theorem 3.10). The following two results (which are proved in Sections 4 and 5) isolate a new class of systems defined in terms of fat intersections and situated strictly between weak and mild mixing. A priori it could happen that for weakly mixing systems the sets R ɛ A,B always belong to the intersection of IP + and D. The following theorem shows that this is not always so. (It also provides a proof that the family D \ IP + is nonempty). Theorem 1.7. There exists a weakly mixing probability measure preserving system (X, B, µ, T ), sets A, B B and ɛ > 0 such that the set R ɛ A,B is not IP +. On the other hand, the requirement that all sets R ɛ A,B are IP + is insufficient for mild mixing (in particular D IP + \ IP is nonempty): Theorem 1.8. There exists a weakly mixing but not mildly mixing probability measure preserving system (X, B, µ, T ), such that all the sets R ɛ A,B are IP + (but not all of them are IP ). Questions. (a) Does there exist a mildly mixing system for which not all sets RA,B ɛ belong to + (c.f. Theorem 2.11 (1)). (b) Does there exist a mildly mixing not mixing system for which all sets RA,B ɛ belong to +? (Here we do not even know whether IP + \ is nonempty.) (c) More generally, what is the dynamical condition equivalent to R(X, B, µ, T ) +? The following figure gives an overview of the classes of systems under study and inclusions between them. The symbol R(F) stands for the class of systems (X, B, µ, T ) such that R(X, B, µ, T ) F.

10 VITALY BERGELSON AND TOMASZ DOWNAROWICZ Section 2. IP-sets, central sets and D-sets in topological dynamics Recall that βz is the Stone-Čech compactification of Z consisting of ultrafilters, which has a natural semigroup structure. On βz there is also the natural action τ which extends the map n n + 1 on Z. If p βz is ultrafilter then the p-limit of a sequence x n of elements of a compact space is defined by the rule p-lim x n = x ( open U x) {n Z : x n U} p. The following fact will be used repeatedly in our paper: if p is an idempotent and T is a continuous selfmap of a compact space then p-lim T n x = y implies p-lim T n y = y (see Proposition 3.2 in [B2]). Every transitive topological dynamical system (X, T ) (with a transitive point x 0 ) is a topological factor of (βz, τ) via the map p p-lim T n (x 0 ) (see e.g. Proposition 7.3 in [E]). The orbit closure of a point x in a topological dynamical system (X, T ) is the set O(x) = {T n x : n Z}. A point x in (X, T ) is recurrent if for every neighborhood U x x there exists n 0 such that T n x U x. It is known ([F], Theorem 2.17) that the set F of return times of a recurrent point x, F = {n Z : T n x U x }, is an IP-set. We also have Theorem 2.1. A set E Z is IP if and only if there exist a compact metrizable dynamical system (X, T ), a pair of points x, y X such that y is recurrent and (y, y)

LARGE SETS OF INTEGERS 11 belongs to the orbit closure of (x, y) in the product system (X X, T T ), and an open neighborhood U (y,y) of (y, y) such that E = {n Z : (T n x, T n y) U (y,y) }. Remark 2. Note that if (y, y) belongs to the orbit closure of (x, y) then x and y are proximal. In general, the conditions that y is recurrent and x is proximal to y do not imply (y, y) O(x, y). For example, x can be a fixpoint in the orbit closure of a recurrent point y x. In order that (y, y) O(x, y) the recurrence of y and the proximality of x and y must be realized along a common sequence of times. Proof of Theorem 2.1. Let y and x be such that y is a recurrent point in X with (y, y) O(x, y) and let U (y,y) be an open neighborhood of (y, y). Consider the set E = {n Z : (T n x, T n y) U y U y }, where U y U y is a product neighborhood of (y, y) contained in U (y,y). It is clear that the set E is infinite, so it contains some s 0. Suppose E contains FS(S), where S is some finite set not containing zero. Let V y U y be an open neighborhood of y such that T s (V y ) U y for all s S. We can find 0 s / S for which (T s x, T s y) V y V y. Then (T s x, T s y) U y U y and (T s +s x, T s +s y) U y U y for every s S. We have shown that E FS(S ), where S = S {s }. By induction, we will obtain a set FS(S) (where S is infinite) contained in E, which proves that E (as well as E) is an IP-set. To prove the converse, consider an arbitrary IP-set E and let x = (x(n)) n Z be the characteristic function of E viewed as an element of the shift system X = {0, 1} Z. Define y = p-lim T n x, where p is an idempotent such that E p (see Definition 1.2(1)). Following the proof of Theorem 3.6 in [B2], we claim that the sequence y starts with the symbol 1: y(0) = 1. By the definition of p-lim, the set R = {n Z : (T n x)(0) = y(0)} belongs to p. So, the intersection R E is nonempty (it belongs to p), which implies that there exists n E with x(n) = y(0). But x(n) = 1 if and only if n E, so y(0) = 1. This implies that E = {n Z : (T n x, T n y) U (y,y) }, where U (y,y) is defined as U 1 X, where U 1 is the cylinder of elements starting with 1. We will now introduce C-sets and D-sets in a similar way, by imposing additional conditions on the recurrence of y. A point y contained in a dynamical system (X, T ) is uniformly recurrent if, for any neighborhood U of y, the set of return times {n Z : T n y U} is syndetic. It is well known that y is uniformly recurrent if and only if the orbit closure O(y) of y is minimal. Central sets have been defined by H. Furstenberg ([F, Def. 8.3]) as follows: Definition 2.2. A set C Z is central if there exists a compact metrizable dynamical system (X, T ), a point x X proximal to a uniformly recurrent point y X and an open neighborhood U y of y such that C = {n Z : T n x U y }.

12 VITALY BERGELSON AND TOMASZ DOWNAROWICZ One can show that C is central if and only if C is a member of a minimal idempotent in βz (see [B-H1] Corollary 6.12 and [B2], Theorem 3.6). We have already used this equivalent form of definition of central in Section 1, Definition 1.2. Central sets can also be characterized with the help of product systems, as follows. Theorem 2.3. A set C Z is central if and only if there exist a compact metrizable dynamical system (X, T ), a pair of points x, y X where y is uniformly recurrent, and such that (y, y) belongs to the orbit closure of (x, y) in the product system (X X, T T ), and an open neighborhood U (y,y) of (y, y) such that C = {n Z : (T n x, T n y) U (y,y) }. Proof. As we have mentioned in Remark 2, even if y is recurrent and x is proximal to y, (y, y) does not have to belong to the orbit closure of (x, y). Nevertheless, it is easy to see that if y is uniformly recurrent then proximality of x and y does imply that (y, y) belongs to the orbit closure of (x, y). This observation is crucial in the proof. Let C be central, and let x and y be as in Definition 2.2. Then (y, y) belongs to the orbit closure of (x, y), and C = {n Z : (T n x, T n y) U (y,y) }, where U (y,y) = U y X. Conversely, if C = {n Z : (T n x, T n y) U (y,y) } with assumptions on x and y as in the formulation of the theorem, then C is central directly by Definition 2.2, using (x, y) and (y, y) as a pair of points in the direct product (X X, T T ). Notice that (y, y) is uniformly recurrent in the product system. Now we focus on D-sets. In the Introduction we have defined them analogously to C-sets by replacing minimal idempotents by a wider class of idempotents all of whose members have positive upper Banach density, so that the class D of D- sets is (strictly) intermediate between IP and C. We are interested in obtaining a characterization of D-sets, analogous to that of IP-sets and C-sets (in terms of visits of (T n x, T n y) to U (y,y) ) by imposing on y an appropriate type of recurrence condition, as defined below. Definition 2.4. A point y contained in a (not necessarily metrizable) dynamical system (X, T ) is essentially recurrent if the set of visits {n Z : T n y U y } for any neighborhood U y of y has positive upper Banach density. Obviously, since every syndetic set has positive upper Banach density, every uniformly recurrent point is essentially recurrent. A characterization of essentially recurrent points in terms of the properties of their orbit closures is provided below.

LARGE SETS OF INTEGERS 13 Definition 2.5. A dynamical system (Y, T ) will be called measure saturated if the union of the topological supports of all invariant probability measures 6 carried by Y is dense in Y. In other words for every nonempty open set U there exists an invariant measure µ such that µ(u) > 0. Note that every minimal system is measure saturated. Theorem 2.6. A point y is essentially recurrent if and only if the orbit closure O(y) is measure saturated. Proof. First let us show that if a point y is essentially recurrent then its orbit closure is measure saturated. Let U y y be an open set and let U y be open and such that U U y. Since y is essentially recurrent, the set A = {n Z : T n y U} has positive upper Banach density d. Let I n be a sequence of intervals in Z, with I n (as n ) such that the ratios A I n I n converge to d. Let µ n (n = 1, 2,... ) be the normalized counting measures supported by the sets {T i y : i I n }, and let µ be a weak accumulation point 7 of µ n. Clearly, µ is T -invariant, supported by O(y) and satisfies µ(u) > 0, and thus µ(u y ) > 0. We have proved that the closure M of the union of supports of all invariant measures carried by O(y) contains y. Since M is a closed invariant set, it follows that M = O(y), i.e., O(y) is measure saturated. Conversely, assume that O(y) is measure saturated. Let U y y be an open set. Then there exist an invariant measure µ supported by O(y) such that µ(u y ) > 0. The ergodic theorem assures that the function 1 f(x) = lim n n n 1 Uy (T i (x)) satisfies f dµ = µ(u y ) > 0. Thus there exists y O(y) with f(y ) = d > 0. In other words, the set R = {n Z : T n y U y } has natural density d, i.e., satisfies R [1,n] lim n n = d. Note now that for any m N there exists n Z such that for any i [0, m], T n+i (y) U y if and only if T i (y ) U y. It follows that the set {n Z : T n y U y } has positive upper Banach density (at least d) and hence y is essentially recurrent. Definition 2.7. Let p be an idempotent in βz. We will call p essential if every member of p has positive upper Banach density. We are in a position to provide a dynamical definition of D-sets, which is completely analogous to the characterizations of IP-stes and central sets. 6 The classical Bogoliubov-Krylov Theorem guarantees the existence of at least one invariant probability measure. The topological support of a probability measure is the smallest closed set of measure 1. 7 A sequence of measures µ n converges to µ weakly if R f dµ n R f dµ for every continuous function f on the space Y. i=1

14 VITALY BERGELSON AND TOMASZ DOWNAROWICZ Theorem 2.8. A set D Z is a D-set if and only there exists a compact metrizable dynamical system (X, T ), points x, y X with y essentially recurrent, for which the orbit closure of (x, y) in the product system (X X, T T ) contains (y, y), and an open neighborhood U (y,y) of (y, y) such that D = {n Z : (T n x, T n y) U (y,y) }. Before we prove the theorem we need a series of lemmas. Lemma 2.9. An idempotent q βz is an essentially recurrent point in (βz, τ) if and only it is essential. Remark 3. Glasner [G] introduces a set Z in βz defined as the closure of supports of all invariant measures on βz and he proves that it is a so-called kernel for the family of sets of positive upper Banach density. In fact one implication of the above lemma could be deduced from that result, but we choose to give an independent proof. Proof of Lemma 2.9. Let q be essentially recurrent and let E be any element of q. The closure E of E in βz can be interpreted as a neighborhood of q. There exists an invariant measure µ such that µ(e) > 0. Since µ is supported by the orbit closure of 0, the set of visits of 0 to this neighborhood (which is E) has positive upper Banach density (the same argument as in the proof of Theorem 2.6 applies). The converse is also true. The map p p + q is a factor map from βz onto O(q) and both 0 and q map to q. A neighborhood U q of q in O(q) lifts to a neighborhood V q of q in βz and the set R q of times of visits of q in U q contains the set R 0 of the times of visits of 0 in V q. But the set R 0 is a member of q (because its complement is not). Since q is assumed to be an essential idempotent, all members of q have positive upper Banach density (see Definition 2.7.). It follows that R 0 has positive upper Banach density and hence, by Definition 2.4, q is essentially recurrent. It is obvious that if π : X Y is a topological factor map and y Y is uniformly recurrent then there exists a uniformly recurrent π-lift x X of y (because the preimage of O(y) is invariant and any one of its minimal subsets must map onto O(y)). The lemma below is an analogous statement for essentially recurrent points. Lemma 2.10. Let π : X Y be a topological factor map (surjection) between dynamical systems (X, S) and (Y, T ). If y is an essentially recurrent point in Y then there exists an essentially recurrent π-lift x of y. Moreover, we can find such x for which O(x) contains no proper closed invariant subset mapping by π onto O(y). Proof. Applying Zorn s Lemma to the family of all lifts of O(y), i.e., of closed invariant sets mapping by π onto O(y), we can find a minimal such lift X 0 X. Let x be any lift of y contained in X 0. Since O(x) X 0 and it maps onto O(y), by minimality O(x) = X 0. On the other hand, since every invariant measure carried

LARGE SETS OF INTEGERS 15 by O(y) lifts to at least one invariant measure carried by O(x), the closure X 1 of the union of the supports of all invariant measures carried by O(x) maps onto a closed set containing the union of supports of all invariant measures carried by O(y). Since y was assumed essentially recurrent, X 1 maps onto O(y) and hence, as a closed invariant subset of X 0 it also equals X 0. This proves that x is essentially recurrent, and that its orbit closure is a minimal lift of O(y), as required. Proof of Theorem 2.8. Let D = {n Z : (T n x, T n y) U (y,y) }, where x and y are as in the formulation of the theorem. Consider a factor map π : βz O(x, y) defined by p π(p) := p-lim (T n x, T n y). By assumption, O(y, y) O(x, y). Since O(y, y) is contained in the diagonal, it is topologically conjugate to O(y) and hence (y, y) is essentially recurrent. By Lemma 2.10, we can find in βz an essentially recurrent π-lift p 1 of (y, y) whose orbit closure is a minimal lift of O(y, y). We will show that p 1 can be replaced by an idempotent. Consider the set I = {p O(p 1 ) : π(p) = (y, y)}. By an elementary verification, I is a closed semigroup of βz, so it contains an idempotent q. Since q maps to (y, y), its orbit closure maps onto O(y, y). By minimality of the lift O(p 1 ), q has the same orbit closure as p 1, and hence is essentially recurrent. Finally, D q follows from the two facts: 1) (T n x, T n y) does not belong to the neighborhood U (y,y) of (y, y) for all n D c. 2) q-lim (T n x, T n y) = (y, y). This implies D c / q, so that D must belong to q. We have completed the proof of one implication. We now proceed to prove the converse implication of the theorem. Let D be a D-set (i.e., a member of an essentially recurrent idempotent). Identify the characteristic function of D with a point x in {0, 1} Z and denote the action of the shift transformation by T. Define y = q(x) := q-lim T n x. Since q is an idempotent, q(y) = y, so q(x, y) = (y, y), i.e., the orbit of (x, y) accumulates at (y, y), as required. in the theorem. Now we repeat the argument used in the proof of Theorem 2.1: The set R = {n Z : (T n x)(0) = y(0)} belongs to q, so R D. Since x(n) = 1 for n D, y(0) = 1. As a consequence, D = {n Z : (T n x, T n y) U (y,y) }, where U (y,y) = U 1 X, where U 1 denotes the cylinder of elements starting with 1 and X denotes the full shift space. The last thing we need is to verify that y is essentially recurrent. But this is immediate, because y is the image of q via the factor map π : βz O(x) given by p p(x), and it is elementary to see that any factor map preserves essentially recurrent points. Remark 4. Note that if y is an essentially recurrent point in the orbit closure of x and x, y are proximal, then the set {n Z : T n x U y } need not be a D- set. For example, let y = (y(n)) be a forward transitive point in the full shift on three symbols 0,1,2 (such y is essentially recurrent) with y(0) = 0 and let

16 VITALY BERGELSON AND TOMASZ DOWNAROWICZ x be as follows: x(n) = 1 whenever y(n) = 1 (this makes x and y proximal), x[m, n] = y[0, n m] if y[m, n] = 222...2 and y(m 1) 2 (then x is forward transitive, hence its orbit closure contains y), and x(n) = 2 whenever y(n) = 0. Then the set {n Z : T n x U y } is not even an IP-set: If p-lim T n (x) = y then p-lim T n (y) y (p-lim T n (y) has the symbol 2 at zero coordinate), and hence p is not an idempotent. We will now focus on the dual families, more precisely, on proving the noncontainment claims formulated in the introduction below the main diagram. Theorem 2.11. 1. There exists an IP set which is not +. 2. There exists a C -set which is not D +. Proof. A set of integers enumerated increasingly as (a n ) (over n Z or n N) is said to have progressive gaps if it contains a subsequence a nk (we will call each finite subset {a nk +1, a nk +2,..., a nk+1 } a chunk) such that for n k +1 < i n k+1 one has a i a i 1 > a nk+1 a i (inside each chunk every gap is larger than the distance to the right end of the chunk) and a nk +1 a nk (the gaps between the chunks tend to infinity). A structure of a set with progressive gaps is shown below: chunk {}}{........................ a n1 a n2 a n3 A typical example of a set with progressive gaps is the difference set (S) for a rapidly (for example exponentially) increasing sequence S. It is not hard to see that in such a set, for any fixed d, the set of elements a i, such that there exists j > i with a j a i = d, is either finite or its gaps tend to infinity (because the distance d can eventually occur only inside the chunks and then only once in every chunk). Notice the following property of all IP-sets F : a certain distance d between elements of F appears along an IP-set. Indeed, if F contains the set of finite sums FS(S) with S = (s n ) then the distance s 1 occurs between all pairs b and s 1 + b for every b FS(A ), where A = (s n ) n 2. Clearly, analogous statement holds for shifted IP-sets: certain distance d occurs along a shifted IP-set. In particular, the gaps between pairs with distance d do not tend to infinity. We conclude that a set with progressive gaps does not contain any shifted IP-set. Let (r k ) k 1 be a sequence containing all integers. Using the above observation we will now describe how to construct a set E as a union over all integers k of shifted by r k -sets E k such that E has progressive gaps, hence contains no shifted IP-sets. Clearly, the complement of such a set E is IP and not +. Begin with the difference set of a rapidly growing sequence, so that it has progressive gaps. Let E 1 be this difference set shifted by r 1. Inductively, suppose a union of k

LARGE SETS OF INTEGERS 17 shifted (by r 1,..., r k ) difference sets makes a set E k with progressive gaps. We will now create a new difference set (S) with progressive gaps, whose chunks fit into the large gaps of E k r k+1, in such a way that E k+1 defined as E k ( (S) + r k+1 ) maintains progressive gaps. Let s 1 = 1. Suppose we have defined s 1,..., s n S. This determines a part of (S) and the shape of the next chunk {s n+1 s n,..., s n+1 s 1 }. The next element s n+1 of S determines only the shifting of this new chunk. By an appropriate choice of s n+1 we can position this chunk in the central part of some very large gap between the chunks of E k r k+1. In the union (E k r k+1 ) (S) this gap splits into two gaps about half the original size with a new chunk in the middle. Similarly we choose s n+2, and on, until the whole sequence S is defined. It is clear that (E k r k+1 ) (S) (and hence E k ( (S) + r k+1 )) maintains progressive gaps. We can pass to step k + 2 and further steps. If in each step k we split only gaps larger than some increasing (with k) threshold value, the set E = k E k will maintain progressive gaps, and it is a union of shifted -sets, as needed to complete the proof of the statement (1). We now describe the construction of a C sets which is not D +. The idea is the same as in the preceding argument, except that we will use different properties of sets. Suppose there exists a non-piecewise syndetic set E such that E + k is a D-set for each k Z. Such E contains no shifted C-sets (recall that C + = PS). Thus the complement of E is a C -set, and since every shift of E misses a D-set, it is not a D +-set. It remains to construct such non-piecewise syndetic set E. Consider a topologically weakly mixing 8 and measure saturated system (X, T ) with the property that the closure of the union of all minimal sets is smaller than X. An explicit construction of such an example is provided in the Appendix (the example is in fact topologically mixing, with an invariant measure having full support, and with a fixpoint as the unique minimal set). Let U be an open set disjoint from another open set V containing the union of all minimal sets. Notice that the orbit closure of y is conjugate to that of (y, y) in the product system. If y is a transitive point then it is essentially recurrent, and so is (y, y). There exists a pair (x, y) transitive in X X with both x and y contained in U. Then, for any integer k, the pair (T k x, y) is also transitive, hence its orbit closure contains (y, y). Thus the set {n k : T n x U} is a D-set (write it as {j : (T j T k x, T j y) U X}). This implies that any shift of the set E = {n Z : T n x U} is a D-set, as required. This set E is not piecewise syndetic; if it was we could easily construct a uniformly recurrent point in the closure of U, which is impossible, since all such points are in V. 8 A topological dynamical system (X, T ) is said to be topologically weakly mixing (mixing) if for any nonempty open sets A, B X the set {n Z : T n A B } is thick (cofinite).

18 VITALY BERGELSON AND TOMASZ DOWNAROWICZ Section 3. Applications of the dual families to unitary and measure preserving actions. This section contains proofs of the nontrivial implications in Theorem 1.6. We begin with the role of the and +-sets. Theorem 3.1. (see Theorem 1.6 (1)). In every measure-preserving system the set R ɛ A,A of fat intersections for one set A is. Proof. (cf. [B1], p. 49; see also [K-Y] Proposition 4.1) First observe that if A n is any sequence of sets of equal measures α in a probability space, then for every ɛ > 0, the inequality µ(a i A j ) > α 2 ɛ holds for at least one pair of indices i < j. Indeed, suppose the contrary and consider the function n i=1 1 A i. Its inner product with 1 equals nα, while the square of its L 2 -norm is easily seen to be at most n 2 (α 2 ɛ) + n. For large n this contradicts the Cauchy-Schwarz inequality. Once this is established, take any injective sequence S = (s n ) and let A n = T s n A. Then µ 2 (A) ɛ < µ(t s i A T s j A) = µ(a T s j s i A) holds for at least one pair of indices i < j, proving that RA,A ɛ intersects (S). Remark 4. We remark that the above proof actually shows that RA,A ɛ has nonempty intersection with every large enough finite difference set. Theorem 3.2. (see Theorem 1.6 (2)). Let (X, B, µ, T ) be an ergodic rotation of a compact abelian group (where µ is the Haar measure). Then, for any A, B B and ɛ > 0 the set R ɛ A,B is +. Proof. The proof is based on a simple observation, that for group rotations Khintchine s theorem takes on a stronger form. Namely, if (X, B, µ, T ) is a (not necessarily ergodic) compact abelian group rotation, then for any C B and ɛ > 0, one actually has that the set R ɛ C = {n Z : µ(c T n C) µ(c) ɛ} is (note that in the displayed formula we have µ(c) rather than µ(c) 2 ). For, let (S) = {s i s j } where S = (s i ). Finding a subsequence s ik such that T s i k (e) converges we obtain a uniformly convergent sequence of maps T s i k. Thus T s i k s il converge to identity uniformly (hence strongly in L 1 (µ)), which implies that µ(c T n C) µ(c) ɛ for some n of the form s ik s il, (belonging to (S)). Returning to the ergodic case and two sets A, B B, let us first find (by ergodicity) an integer n 0 such that µ(a T n 0 B) > µ(a)µ(b) ɛ 2. Denoting C = A T n 0 B one easily obtains that R ɛ A,B R ɛ 2 C + n 0, which implies the assertion. We will discuss now the connections between essential idempotents and unitary actions. Consider a unitary operator U on a separable Hilbert space H. We will

LARGE SETS OF INTEGERS 19 use the orthogonal decomposition H = H c H wm, where { } H c = x H : {U n x} n Z is compact in the norm topology, { } N 1 1 H wm = x H : U n x, x 0 N n=0 (see [Kr], Section 2.4 and [B2], Theorem 4.5). Recall that in a Hilbert space the norm convergence lim x n = y is equivalent to the conjunction of the weak convergence of x n to y and the convergence of norms lim x n = y. Since any unitary operator U is an isometry, the relation p-lim U n x = x for some p βz holds in the weak topology if and only if it holds in the strong topology. Lemma 3.3. If p βz is an idempotent then for any x H c one has p-lim U n x = x. Proof. By definition of H c, U acts on the compact metric space {U n x} n Z where it is distal (it is actually an isometry). In distal systems one has p-lim U n x = x for any idempotent (if p-lim U n x = y for an idempotent p then also p-lim U n y = y hence x and y are proximal, and so, by distality, x = y). The above statement can be reversed for essential idempotents: Lemma 3.4. If p βz is an essential idempotent and p-lim U n x = x for some x H then x H c. Proof. For ɛ > 0 consider the set E = {n Z : U n x x < ɛ 2 }. Clearly E p. Note that for any n 1, n 2 E one has T n 1 n 2 x x = T n 1 x T n 2 x T n 1 x x + T n 2 x x < ɛ. Since E p, it has positive upper Banach density, which implies that E E is syndetic (see [F], Prop. 3.19(a) or [B1] p. 8), i.e., finitely many shifted copies of E E cover Z. This in turn implies that finitely many preimages of the ɛ-ball around x cover the orbit of x. Since U is an isometry we have covered the orbit by finitely many ɛ-balls, hence the orbit of x is precompact, i.e., x H c. Lemma 3.5. If p βz is an essential idempotent then for any x H wm one has p-lim U n x = 0 weakly. Proof. By compactness of the ball of radius x around zero in the weak topology, there exists some y such that p-lim U n x = y weakly. Since H wm is invariant and closed, y H wm. On the other hand, p is an idempotent, so p-lim U n y = y. By Lemma 3.4, y H c. This implies y = 0. Recall that a unitary operator U acting on a Hilbert space H is called weakly mixing if it has no non-trivial eigenvectors. One can show that U is weakly mixing

20 VITALY BERGELSON AND TOMASZ DOWNAROWICZ if and only if in the decomposition H = H c H wm one has H c = {0} (see [Kr], Thms 3.4 and 4.4). Let now (X, B, µ, T ) be an invertible weakly mixing system. It is not too hard to check that in this case the unitary operator induced by T on L 2 (µ) is weakly mixing in the above sense on the orthocomplement of the space of constant functions. This leads to the following corollary of Lemmas 3.4 and 3.5: Corollary 3.6. An invertible probability measure-preserving system (X, B, µ, T ) is weakly mixing if and only if for every f L 2 (X) and any essential idempotent p βz, p-lim T n f = f dµ (in the weak topology). Equivalently, (X, B, µ, T ) is weakly mixing if and only if for any A, B B and any essentially recurrent idempotent p, p-lim µ(a T n B) = µ(a)µ(b). We now turn our attention to the D -sets. It was proved in [B2], Theorem 4.4 that a unitary operator U acting on a Hilbert space H is weakly mixing if and only if for any ɛ > 0 and any pair x, y H the set R ɛ x,y = {n Z : U n x, y > ɛ} is C. We will show that a slight modification of this proof provides a somewhat stronger result. Theorem 3.7. A unitary operator U acting on a Hilbert space H is weakly mixing if and only if for any ɛ > 0 and any pair x, y H the set R ɛ x,y is D. Proof. If U is weakly mixing then H c = {0} and the result follows from Lemma 3.5. Assume now that for any ɛ and x, y H the set Rx,y ɛ is D. If U is not weakly mixing then there exists x H c, x 0. By Lemma 3.3 one has p-lim U n x = x for any essential idempotent p. One has then p-lim U n x, x = x 2, which implies that for small enough ɛ > 0 the set Rx, x ɛ is not D. We can now continue with proving the statements of Theorem 1.6: Theorem 3.8. (see Theorem 1.6 (3)). An invertible probability measure-preserving system (X, B, µ, T ) is ergodic if and only if for any A, B B and ɛ > 0 the set R ɛ A,B belongs to D +. Proof. (cf. [B2], Theorem 4.11) Assume that (X, B, µ, T ) is ergodic. Denote f = 1 A and g = 1 B. Decompose g = g 1 +g 2, g 1 H c, g 2 H wm. Note that g 1 dµ = µ(b). By the von Neumann Ergodic Theorem, 1 N N 1 n=0 f(t n x)g 1 (x) dµ(x) f dµ g 1 dµ = µ(a)µ(b),