The structure of unitary actions of finitely generated nilpotent groups

The structure of unitary actions of finitely generated nilpotent groups A. Leibman Department of Mathematics The Ohio State University Columbus, OH 4320, USA e-mail: leibman@math.ohio-state.edu Abstract Let G be a finitely generated nilpotent group of unitary operators on a Hilbert space H. We prove that H is decomposable into a direct sum H = α A L α of pairwise orthogonal closed subspaces so that elements of G permute the subspaces L α, and if T(L α ) = L α, then the action of T on L α is either scalar or has continuous spectrum. We also provide examples showing that analogous results do not hold for solvable non-nilpotent groups. 0. Introduction Let H be a Hilbert space and let G be a group of unitary operators on H. Then H is the sum of two G-invariant orthogonal subspaces: H = H c (G) H wm (G) such that G is weakly mixing on H wm (G) and has discrete spectrum on H c (G). The space H c (G) is spanned by finite-dimensional G-invariant subspaces of H, and consists of vectors whose orbits under the action of G are precompact: H c (G) = { u H Gu is precompact }. We will say that G is compact on H c (G). The space H wm (G) is the maximal G-invariant subspace M of H such that for any unitary action of G on a Hilbert space N, the space M N does not contain nonzero elements which are invariant with respect to the G-action (defined on M N by T(u v) = Tu Tv, T G). If G is an amenable (in particular, abelian or nilpotent) group, then H wm (G) can also be described as the maximal subspace of H such that for any ε > 0 and any u H wm (G) and v H, the set { T G Tu,v > ε } has zero density in G (with respect to a F/olner sequence). Now assume that we are interested in weakly mixing/compact properties of individual elements of G. For a unitary operator T on H, denote by H wm (T) the space { u H the set { n Z T n u,v < ε } } has zero density in Z for any v H, Supported by NSF grant DMS-9706057

and by H c (T) the space { u H the set { T n u n Z } } is precompact. Then H c (T) is the subspace of H generated by eigenvectors of T, H wm (T) is the maximal subspace of H where T has continuous spectrum, and H = H c (T) H wm (T) ([KN]). Now, for T G, the decomposition H = H c (G) H wm (G) gives very little information about H c (T) and H wm (T): one can only claim that H c (G) H c (T) and hence, H wm (T) H wm (G). It may even happen that G is weakly mixing on H while all its elements are compact on H. This is not the case, however, when G is a finitely generated abelian group: if the generators of G are compact on H, then G itself is compact on H. For such G, the following structure theorem can be formulated: Theorem A. Let G be a finitely generated abelian group of unitary operators on a Hilbert space H. Then H is a direct sum H = γ Γ H γ of G-invariant closed subspaces such that for every γ Γ, there is a subgroup H γ of G whose action is scalar on H γ, and any T G\H γ is weakly mixing on H γ. Proof. Assume that there is T G which is neither scalar nor weakly mixing on H. Let H be an eigenspace of T ; then H is a closed G-invariant proper subspace of H. If there is T 2 G which is neither scalar nor weakly mixing on H, pick a nontrivial eigenspace H 2 H for T 2, and so on. Since the subgroup of G which is scalar on H k+ is bigger than the subgroup of G which is scalar on H k, and since G satisfies the ascending chains condition, this process can not be infinite. As a result we get a closed G-invariant subspace H K such that every T G is either scalar or weakly mixing on H K. Passing to H K and applying the Zorn lemma completes the proof. Theorem A, despite its triviality, is a useful tool in proving ergodic theorems involving several commuting unitary operators: it allows to reduce the problem to the situation where any product of the given operators is either scalar or weakly mixing. The aim of this paper is to generalize Theorem A to the case where G is a finitely generated nilpotent group; counterexamples in 3.4 suggest that the class of nilpotent groups is the natural domain to which this theorem can be extended. We prove the following: Theorem N. Let G be a finitely generated nilpotent group of unitary operators on a Hilbert space H. Then H is representable as the direct sum of a system {L α } α A of closed pairwise orthogonal subspaces so that (a) elements of G permute these subspaces: for any T G and α A one has T(L α ) = L α for some α A, and (b) if T preserves L α, T(L α ) = L α, then either T is scalar on L α or T is weakly mixing on L α. 2

Some fragments of our structure theory for unitary actions of nilpotent groups were obtained and utilized in [L] and [BL]. As a direct corollary of Theorem N let us mention the following generalization of the classical von Neumann theorem, proved in [BL]: if unitary operators T,...,T t on a Hilbert space H generate a nilpotent group, then N for any polynomials p,...,p t Z[n] and any u H, lim N N n= Tp (n)...t p t(n) t u exists in H. In [L], the structure theory was developed for the (more complicated) case of nilpotent groups of unitary cocycles on a Hilbert bundle, and for nilpotent groups of measure preserving transformations of a measure space. In this paper we present the structure theorem in full detail and in a self-contained way in the purely Hilbertian situation. In particular, we are avoiding here the usage of the apparatus of G-polynomials, used in [L] and [BL]. The structure of the paper is as follows. In Section we collect necessary background information about nilpotent groups and introduce some notation. In Section 2 we define what we call a primitive action of a nilpotent group of unitary operators on a Hilbert space and study its properties. Finally in Section 3 we show that any unitary action of a finitely generated nilpotent group is reducible to primitive actions. We also provide examples which show that the results of this paper do not extend to polycyclic non-nilpotent groups. Acknowledgment. I thank V. Bergelson for his support, encouragement and help. I also thank the referee for corrections and useful suggestions.. Notation and background.. A group G is called nilpotent if it possesses a finite central series, that is, a sequence of normal subgroups { G } = G 0 G... G k = G such that for each i =,...,d, G i /G i is in the center of G/G i. The minimal length d of a central series in G is called the nilpotency class of G. We will need the following facts (for proofs see [KM] and [H], or [BL]):.2. Lemma. Any subgroup and any factor group of a nilpotent group are nilpotent. Any subgroup of a finitely generated nilpotent group is finitely generated..3. Lemma. Let G be a finitely generated nilpotent group. Then any nondecreasing sequence H H 2... of its subgroups stabilizes..4. Lemma. If G is a finitely generated nilpotent group such that all its generators have finite orders, then G is finite..5. Lemma. Let G be a finitely generated nilpotent group. Then there is a finite set S,...,S s G such that for every T G there are a,...,a s Z such that T = S a...sa s s..6. Let H be a subgroup of a group G. We will call the set of elements of G having finite 3

order modulo H the closure of H (in G) and denote it by H: We will say that H is closed if H = H. H = { T G T n H for some n N }..7. Lemma. If G is a nilpotent group and H is its subgroup, then H is a closed subgroup of G. If G is finitely generated, then H has finite index in H..8. For a subgroup H of a group G we denote by N(H) the normalizer of H in G: N(H) = {T G T HT = H}. We also define N 0 (H) = H, N k (H) = N ( N k (H) ), k =,2,....9. Lemma. If G is a nilpotent group of nilpotency class d, then for any subgroup H of G, N d (H) = G..0. Lemma. If H is a closed subgroup of a nilpotent group G, then N(H) is also closed in G... Let a group G act on a set V. For v V, we will denote its orbit {Tv} T G by Gv. The stabilizer Stab(v) of v V is the subgroup {T G Tv = v} of G..2. Lemma. Let a nilpotent group G act on a set V, let T,...,T t generate G, and let for some v V all sets {T n v} n Z,...,{T n t v} n Z be finite. Then the orbit Gv is finite..3. Now, let G be a group of unitary operators on a Hilbert space H. Given u H, we will say that G is scalar on u if Tu = λ(t)u with λ(t) C for all T G. By abuse of language, we will say that G is finite on u if the set Gu is finite, and that G is compact on u if Gu is precompact in H. If L is a subspace of H, we will say that G is scalar, finite or compact on L if G is, respectively, scalar, finite or compact on every u L. We define H f (G) = { u H G is finite on u }, and H c (G) = { u H G is compact on u }..4. Let T be a unitary operator on H. We will say that T is scalar, finite or compact on u H if the group {T n } n Z is, respectively, scalar, finite or compact on u. If T is not finite on u, that is, if the set {T n u} n Z is infinite, we will say that T is totally ergodic on u. N We will say that T is weakly mixing on u if lim N N n= T n u,v = 0 for all v H. If L is a subspace of H, we will say that T is scalar, finite, compact, totally ergodic or weakly mixing on L if T is, respectively, scalar, finite, compact, totally ergodic or weakly mixing on every u L. 4

.5. Lemma. Let T be a unitary operator on a Hilbert space H. Assume that T is weakly mixing on u H, and let a sequence {v(n)} n N be precompact in H. Then T n u,v(n) = 0. N lim N N n= Proof. Fix any ε > 0, and let v,...,v k be an ε-net for the set { v(n) n N }. Let N = C... C k be a partition of N such that v(n) v j < ε whenever n C j. Then lim N N N T n u,v(n) n= = lim N N lim N N lim N N n N n C n N n C T n u,v(n) +...+ lim T n u,v +...+ lim N T n u,v +...+ lim n= Since ε is arbitrary, the statement follows. N.6. For a unitary operator T on H, we define N N N N N n N n C k n N n C k T n u,v(n) T n u,v k +ε u N T n u,v k +ε u ε u. N H f (T) = { u H T is finite on u }, H c (T) = { u H T is compact on u }, and H wm (T) = { u H T is weakly mixing on u }. It follows from Lemma.5 that H wm (T) H c (T). The following fact is well known (see [KN]): Theorem. Let T be a unitary operator on a Hilbert space H. Then H c (T) is the subspace generated by eigenvectors of T and H wm (T) = H c (T)..7. For u H, we define G c (u) = { T G T is compact on u }, and G f (u) = { T G T is finite on u }. If L is a subspace of H, we also define G c (L) = { T G T is compact on L }, and G f (L) = { T G T is finite on L }..8. Lemma. Let G be a nilpotent group of unitary operators on a Hilbert space H and let u H. Then G f (u) is a subgroup of G. n= Proof. G f (u) = Stab(u), which is a subgroup of G by Lemma.7. 5

.9. Lemma. Let G be a nilpotent group generated by unitary operators T,...,T t on a Hilbert space H. Then H f (G) = t H f (T) = H f (T i ). Proof. This follows from Lemma.2. T G Statements analogous to Lemma.8 and Lemma.9 hold for G c and H c as well, but we will be able to show this only in the end of the paper (see Corollaries 3.2 and 3.3 below)..20. Finally, the last piece of notation: If H is a Hilbert space and {L α } α A is a system of pairwise orthogonal subspaces of H, then by α A L α we will understand the closure of the direct sum of the subspaces L α. i= 2. Primitive actions 2.. Definition. Let G be a finitely generated nilpotent group of unitary operators on a Hilbert space H. We will say that the action of G is primitive if H is the direct sum of a system {L α } α A of closed pairwise orthogonal subspaces of H so that: (a) G transitively acts on the set of indices A so that T(L α ) = L Tα for T G, α A. (b) For α A, let H α G be the stabilizer of α. Then every element of H α is either scalar on L α, or weakly mixing on L α. We will call the decomposition H = α A L α a primitive decomposition of H. 2.2. Example. Here is an example of a primitive action of a nilpotent group. Let {u i,j } i,j Z be an orthonormal basis in a Hilbert space H. Let S, P and T be the operators on H defined by Su i,j = u i,j+, Pu i,j = u i,j+i and Tu i,j = u i+,j. The group H generated by S,P,T is nilpotent: S commutes with both P and T, and [P,T] = P T PT = S. Let L i,j = Span(u i,j ), then H = i,j Z L i,j is a primitive decomposition of H: elements of H permute the subspaces L i,j, the stabilizer of L i,j under this action is the subgroup H i,j generated by PS i, and the action of H i,j on L i,j is trivial. 2.3. Let us concentrate on primitive actions. Through this section we will assume that G is a finitely generated nilpotent group of unitary operators on a Hilbert space H, and that the action of G on H is primitive; let {L α } α A be the corresponding system of subspaces of H. 2.4. For α A, let H α G be the stabilizer of α: H α = { P G Pα = α }. Then T(L α ) = L α for T H α, and T(L α ) L α for T G\H α. DenotebyE α thesubgroupofh α consistingofoperatorswhicharescalaronl α : E α = { P Hα P Lα = λid Lα, λ C }. Also, let F α = { P H α P Lα = e 2πir Id Lα, r Q }. 6

2.5. Lemma. {H α } α A, {E α } α A and {F α } α A are classes of conjugate subgroups in G. Proof. Let α,α A, let T G be such that Tα = α. Then T(L α ) = L α, H α = TH α T, E α = TE α T and F α = TF α T. 2.6. It is clear that E α and F α are normal subgroups of H α. It is also easy to see that E α and F α are closed in H α. Indeed, if T H α \E α, then T preserves L α and is weakly mixing on L α. So, for any n 0, T n is also weakly mixing on L α, and thus T n E α. Also, if T E α \F α, then T n F α for all n 0. 2.7. Lemma. For any α A, G c (L α ) = E α, G f (L α ) = F α, and every T G \ E α is weakly mixing on L α. Proof. If T E α then for some n N, T n is scalar on L α, and thus T is compact on L α. Let T G\E α. If T H α, then T n H α for all n 0, so T n L α L α for all n 0, and so, T is weakly mixing on L α. If T H α \E α, then T n H α \E α for some n N, so T n is weakly mixing on L α, and thus T is weakly mixing on L α. Analogously, since all elements of F α are finite on L α, every T F α is finite on L α. And if T E α \F α, then T n E α \F α for some n N, and then T nm u u for all m 0 and u L α. 2.8.ThefollowingexampleshowsthatthesubgroupsE α andf α arenotnecessarilynormal in E α and F α. As a result, elements L α and L α of the primitive decomposition that have coinciding compact and finite subgroups G c (L α ) and G f (L α ) may have different scalar and rational scalar subgroups E α and F α. Example. Let vectors u,u 2,u 3,u 4 form an orthonormal basis of a (4-dimensional) space H, let T and S be operators on H that permute the vectors of the basis in the following way: u u 2 T T u 3 u 4 S Let G be the group generated by T and S; one can easily check that G is nilpotent ( G = 8, and thus G is nilpotent as a finite p-group). Put L i = Span(u i ), i =,2,3,4, then H = 4 i= L i is a primitive decomposition of H corresponding to G. We have F = F 2 = E = E 2 = S (the group generated by S) and F 3 = F 4 = E 3 = E 4 = TST. The group G itself is finite on H, so E i = F i = G for i =,2,3,4, and E i,f i are not normal in G. 2.9. Now, instead of subspaces L α, we will consider some finite sums of these subspaces. As a result, we will lose the information about subgroups of G whose actions are scalar on elements of the new primitive decomposition, but get better properties of the subgroups of G which are compact or finite on these subspaces. 7

2.0. Proposition. Let G be a finitely generated nilpotent group of unitary operators on a Hilbert space H. Assume that the action of G on H is primitive. Then H is representable as a direct sum of pairwise orthogonal closed subspaces {L β } β B so that: () G acts transitively on the set of indices B so that T(L β ) = L Tβ for any T G and β B. (2) For β B, let H β be the stabilizer of β in G. Then every T H β acts on β without finite cycles: T n β β for all n 0, and so, T n L β L β for all n 0. (3) For β B, H β contains normal subgroups E β and F β such that F β is finite on L β, E β is compact on L β, every P E β \F β is totally ergodic on L β, and every T H β \E β is weakly mixing on L β. Proof. Let H = α A L α be a primitive decomposition of H. Let α A. Then for T G and α = Tα, H α = TH α T = TH α T. Thus if T H α, then H α = H α. Consider the equivalence relation defined on A by α α if α = Tα for some T H α. We have a partition B of A onto pairwise disjoint equivalence classes under : A = β B β, where β = H αα for any α β. Since by Lemma.7, H α is of finite index in H α, all β B are finite. Now, let β B and T G. Take any α β and let α = Tα. Then H α α = TH α T Tα = Tβ, and so, Tβ is the equivalence class β B that contains α. Hence an action of G on B is well defined. This action is transitive since the action of G on A is transitive. () For β B define L β = α β L α H. Clearly, the subspaces L β, β B, span H. Since β B are pairwise disjoint, L β are pairwise orthogonal. And for β B and T G, L Tβ = T(L β ). (2) Let β B. Take any α β. Then Tα β if and only if T H α. So, the stabilizer H β of β in G is H α. Since the subgroup H β = H α is closed in G, the action of G on B has no finite cycles: if T H β, then T n β β for all n 0. (3) Again, let β B and let α β. Since E α,f α H α, E α,f α H α = H β. Now, for any T N(E α ), TE α T = TE α T = E α, so N(E α ) N(E α ). Since H α N(E α ), H α N(E α ). SinceE α isclosed, N(E α )isclosedbylemma.0. So, H β = H α N(E α ). Hence, E α is normal in H β. Analogously, F α is normal in H β. Now, let α β. Then α = Tα for some T H β. Thus E α = TE α T = E α, and, analogously, F α = F α. So, we can put E β = E α and F β = F α for any α β. Then E β and F β are normal subgroups of H β. By Lemma 2.7, for any α β and any u L α, G c (u) = E β, G f (u) = F β and every T H β \ E β is weakly mixing on u. Hence, F β is finite on L β E β is compact on L β, every P E β \F β is totally ergodic on L β and every T H β \E β is weakly mixing on L β. 8

2.. Corollary of the proof. Every L β in Proposition 2.0 is a sum of finitely many elements L α of the primitive decomposition of H. For each β B and α A with L α L β, E β = E α and F β = F α. In particular, E α is a subgroup of finite index in E β, and F α is a subgroup of finite index in F β. For the rest of the section we fix the decomposition H = β B L β described in Proposition 2.0. 2.2. Lemma. There exists M N such that for any β B and any u L β, the cardinality of the set F β u does not exceed M. Proof. Fix any β B. Let α β. The group F α is scalar and finite on L α, thus the cardinality F α u 0 of the orbit F α u 0 does not depend on the choice of (nonzero) u 0 L α. Put M = Ind Fβ (F α ) F α u 0. Then F β u = M for all nonzero u L α. It follows that F β u M for all u L β. Now, let β B and u L β. Choose T G such that Tβ = β. Then Tu L β, and so F β u = T F β Tu = F β Tu M. 2.3. Corollary. There exists M N such that for any β B and any T F β, T M is trivial on L β. 2.4. For u H and β B, let u β be the β-coordinate of u, that is the projection of u onto L β, and let B(u) = { β B u β 0 }. Lemma. For u H, G c (u) = β B(u) E β and G f (u) = β B(u) F β. It follows that G c (u) is a subgroup of G. Proof. For every β B(u) one has G c (u β ) = E β, and it is clear that if T G c (u β ) for all β B(u), then T G c (u). Let T G, T E β for some β B(u). For any β B, T n u β L Tn β, so T n u β is the (T n β)-coordinate of T n u: T n u β = (T n u) Tn β. If T H β, the sequence T n u β, n Z, consists of pairwise orthogonal vectors of equal lengths. If T H β \E β, then T preserves L β and is weakly mixing on L β, so the sequence {T n u β } n Z is not precompact in L β. In both cases, the set { T n u n Z } can not be precompact. In the same way, for any β B(u) one has G f (u β ) = F β. And if T F β for all β B(u), it follows from Corollary 2.3 that T is finite on u. The rest of the proof is analogous to that for H c -spaces. 2.5. Lemma. For any T G, H c (T) = β B T E β L β and H f (T) = β B T F β L β. Proof. This directly follows from Lemma 2.4. 9

2.6. Lemma. For any subgroup H of G, the subspaces T H Hc (T) and T H Hf (T) are H-invariant, T H Hc (T) = β B L β and T H Hf (T) = β B L β. H E β H F β Proof. Let B(H) = { β B H E β }. Since for any β B(H), Eβ is a subgroup of the stabilizer H β of β, B(H) is an H-invariant subset of B. Thus the space β B(H) L β is H-invariant, and it follows from Lemma 2.5 that T H Hc (T) = β B(H) L β. For H f -spaces the proof is completely analogous. 2.7. Lemma. For any subgroup H of G one has H c (H) = T H Hc (T) = β B H E β L β and H f (H) = T H Hf (T) = β B H F β L β. Proof. For H f -spaces the statement follows from Lemma.9 and Lemma 2.6. It is also clear that H c (H) H c (T) for all T H. We only have to check that H c (H) T H Hc (T). Let u T H Hc (T), we will be done if we show that the set Hu is precompact. Fix ε > 0. Let S,...,S s H be elements of G such that for every T H, T = S a...sa s s for some a,...,a s Z (see Lemma.5). The set {S a s s u} as Z is precompact; let V s be a finite ε-net in it. Since by Lemma 2.6 the space T H Hc (T) is H-invariant, all elements of H are compact on elements of V s. Thus the sets {S a s s v} a s Z, v V s, are precompact, and we can find a finite ε-net V s in the set v V s {S a s s v} a s Z. It is easy to see that V s is a 2ε-net for the set {S a s s Sa s s u} as,a s Z. Continuing in this way, we come to a finite set V, which, by the choice of S,...,S s, is an sε-net for Hu. 3. Decomposition of a general action into primitive ones 3.. Now let us return to general, non-primitive actions of a nilpotent group. Here is our main result: Theorem. Let G be a finitely generated nilpotent group of unitary operators on a Hilbert space H. Then H is representable as a direct sum of G-invariant closed subspaces on each of which the action of G is primitive. 3.2. Corollary. Let G be a nilpotent group of unitary operators on a Hilbert space H. Then for any u H, G c (u) is a subgroup of G. Proof. We have to check that, given P,Q G c (u), their product PQ is in G c (u). Thus we can replace G by the group generated by P and Q, and so, assume that G is finitely generated. Let H = γ Γ H γ be a decomposition of H into a sum of G-invariant subspaces such that the action of G is primitive on each of them. For u H, let u γ be the projection of u onto H γ. Then T G is compact on u if and only if T is compact on each u γ. So, G c (u) = γ Γ Gc (u γ ). Lemma 2.4 implies the result. 0

3.3. Corollary. Let G be a nilpotent group of unitary operators on a Hilbert space H, and let T,...,T t generate G. Then H c (G) = T G H c (T) = t H c (T i ). Proof. If T,...,T t are compact on u, then by Corollary 3.2, all elements of G are compact on u. And if T,...,T t are finite on u, then all elements of G are finite on u. Now the result follows from Theorem 3. and Lemma 2.7. 3.4. Counterexamples. Let us now bring two examples that show that the structure theory developed above can not be extended to unitary actions of non-nilpotent groups. Of course, if no restrictions on the group of unitary operators are imposed, one can achieve any effect. Our goal was to find groups which would be maximally close to nilpotent ones, yet for which the statements above would fail. The following examples demonstrate that our theory fails already for polycyclic groups, namely for groups possessing subnormal series with cyclic factors: { G } = G 0 G... G k = G, G i is normal in G i and G i /G i is cyclic for i =,...,k. i= ( ) 0 0 3.4.. Let H = C 2, and let T and S be the unitary operators on H given by T = ( /2 3/2 and S = ). Then the group G generated by T and S is isomorphic to the 3/2 /2 dihedral group D 6, which is polycyclic. However the primitive decomposition described in Definition 2. does not exist for it: H is not a direct sum of two subspaces permuted by T and S. 3.4.2. In our second example the action of a polycyclic group is primitive in the sense of Definition 2., but Corollary 3.2 fails for this action. This shows that the second version of the structure theorem, Proposition 2.0, does not hold in this case. Let {u i,u i } i Z be an orthonormal basis in a Hilbert space H, let T and S be the operators on H that act on the elements of this basis in the following way: u u 0 u T T T S u S u 0 S u S (it is assumed that Su i = u i, i Z). The group G generated by T and S is polycyclic: one can check that the sequence of subgroups S S,TST G is a subnormal series in G. The subspaces L i = Span(u i ) and L i = Span(u i ), i Z, define a primitive decomposition in the sense of Definition 2.. However, G c (L 0 ) = G f (L 0 ) is not a subgroup of G: T,S G c (L 0 ), but ST G c (L 0 ).

3.5. Proof of Theorem 3.. It is enough to find a nonzero G-invariant subspace H of H such that the action of G on H is primitive: then we can pass to the orthogonal complement of H in H, and the Zorn lemma will give the result. Moreover, it is enough to find a closed subspace L of H such that if H is a subgroup of G that preserves L, then T(L) L for all T G \ H, and every P H is either weakly mixing or scalar on L. Indeed, in this case we can put A to be the set of left cosets of H in G, and for α A, L α = TL for any T α; the obtained set {L α } α A defines a primitive decomposition H = α A L α of the G-invariant subspace H = Span { TL T G }. Let K be a maximal subgroup of G for which the subspace L = H c (K) is nontrivial; such a subgroup exists by Lemma.3. K is a closed subgroup: since K is of finite index in K by Lemma.7, H c (K) = H c (K). Let T N(K)\K. Then T(L) = H c (TKT ) = L, so T preserves L. If T had an eigenvector u L, then for the group K generated by K and T we would have u H c (K ), that contradicts the choice of K. Thus, T is weakly mixing on L. Now, let T N 2 (K) \ N(K). Let P K be such that Q = T PT N(K) \ K. Then for any u,v L one has Tu,v = PTu,Pv = N N P n Tu,P n v = N n= N TQ n u,p n v n= = N N Q n u,t P n v. Since Q is weakly mixing on u and the sequence T P n v is precompact, by Lemma.5 N lim N N n= Q n u,t P n v = 0 and so, Tu,v = 0. It follows that T(L) L. Since N(K) is closed by Lemma.0, T n N(K) for all n 0, and thus, T is weakly mixing on L. Now we can use induction on k to show that if T N k (K)\N(K), then T(L) L: find P K for which Q = T PT N k (K) \ K, then Q is weakly mixing on L, and Tu,v N = lim N N n= Q n u,t P n v = 0. It now follows from Lemma.9 that T(L) L for all T G\N(K). Denote the group N(K) by H. Then K is normal in H, T(L) L for any T G\H, T(L) = L for T H, T is compact on L if T K and T is weakly mixing on L if T H \K. The only remaining problem is that K may not be scalar on L. Let E be the subgroup of K which is scalar on L: for P E let P L = µ(p)id L, µ(p) C. Assume that E K. Let { G } = G 0 G... G d = G be a central series in G, let i be such that E G i = K G i and E G i+ K G i+. Pick any S (K G i+ )\(E G i+ ). Then for any T H, the commutator [S,T] = S T ST G i and since K is normal in H, [S,T] K. Thus [S,T] K G i E. Let L L be an eigenspace of S, let S L = λid L. Then for any T H and any u L, STu = TS[S,T]u = λµ([s,t])tu, 2 n=

and so either Tu L (if µ([s,t]) = ), or Tu L. Let H be the subgroup of H that preserves the subspace L. Then T(L ) L for all T H \H and so, for all T G\H. Let E be the subgroup of H which is scalar on M. Then E contains S and thus E is greater than E. Let us replace L by L, H by H, E by E and K by K H. Since this enlarges the subgroup E, by Lemma.3, after several such steps we will come to the position where K = E. Bibliography [BL] V. Bergelson and A. Leibman, A nilpotent Roth theorem, submitted. [H] Ph. Hall, Nilpotent Groups, Queen Mary College Mathematical Notes, 969. [KM] M. Kargapolov and Ju. Merzljakov, Fundamentals of the Theory of Groups, Springer Verlag, 979. [KN] B. Koopman and J. von Neumann, Dynamical systems of continuous spectra, Proc. Nat. Acad. Sci. USA 8 (932), 255-263. [L] A. Leibman, Multiple recurrence theorem for measure preserving actions of a nilpotent group, Geom. and Funct. Anal 8 (998), 853-93. 3