BRANCHING LAWS FOR SOME UNITARY REPRESENTATIONS OF SL(4,R)

BRANCHING LAWS FOR SOME UNITARY REPRESENTATIONS OF SL(4,R) BENT ØRSTED AND BIRGIT SPEH Abstract. In this paper we consider the restriction of a unitary irreducible representation of type A q (λ) of GL(4, R) to reductive subgroups H which are the fixpoint sets of an involution. We obtain a formula for the restriction to the symplectic group and to GL(2, C), and as an application we construct in the last section some representations in the cuspidal spectrum of the symplectic and the complex general linear group. Introduction Understanding a unitary representation π of a Lie groups G often involves understanding its restriction to suitable subgroups H. This is in physics referred to as breaking the symmetry, and often means exhibiting a nice basis of the representation space of π. Similarly, decomposing a tensor product of two representations of G is also an important branching problem, namely the restriction to the diagonal in G G. Generally speaking, the more branching laws we know for a given representation, the more we know the structure of this representation. For example, when G is semisimple and K a maximal compact subgroup, knowing the K-spectrum, i.e. the collection of K-types and their multiplicities, of π is an important invariant which serves to describe a good deal of its structure. It is also important to give good models of both π and its explicit K-types. There has been much progress in recent years (and of course a large number of more classical works, see for example [20], [3], [4], [5]), both for abstract theory as in [8], [9], [], [0], and concrete examples of branching laws in [8], [9], [2], [], [2]. 0 partially supported by NSF grant DMS-007056 Date: September 2, 2008. 99 Mathematics Subject Classification. Primary: 22E47 ; Secondary: F70. Key words and phrases. semisimple Lie groups, unitary representation, branching laws.

2 BENT ØRSTED AND BIRGIT SPEH In this paper, we shall study in a special case a generalization of the method applied in [3] and again in [5]; this is a method of Taylor expansion of sections of a vector bundle along directions normal to a submanifold. This works nicely when the original representation is a holomorphic discrete series for G, and the subgroup H also admits holomorphic discrete series and is embedded in a suitable way in G. The branching law is a discrete sum decomposition, even with finite multiplicities, so-called admissibility of the restriction to H; and the summands are themselves holomorphic discrete series representations for H. Since holomorphic discrete series representations are cohomologically induced representations in degree zero, it is natural to attempt a generalization to other unitary representations of similar type, namely cohomologically induced representations in higher degree. We shall focus on the line bundle case, i.e. the A q (λ) representations. In this case T. Kobayashi [0] obtained necessary and sufficient conditions that the restriction is discrete and that each representation appears with finite multiplicity. Using explicit resolutions and filtrations associated with the imbedding of H in G, we analyze the derived functor modules and obtain an explicit decomposition into irreducible representations. It is perhaps not surprising, that with the appropriate conditions on the imbedding of the subgroup, the class of (in our case derived functor) modules is preserved in the restriction from H to G. Let G be a semisimple linear connected Lie group with maximal compact subgroup K and Cartan involution θ. Suppose that σ is another involution so that σ θ = θ σ and let H be the fixpoint set of σ in G. Suppose that L = L x is the centralizer of an elliptic element x G H and let q = l u, q H = q h be the corresponding θ stable parabolic subgroups. Here we use as usual gothic letters for complex Lie algebras and subspaces thereof; a subscript will denote the real form, e.g. g o. We say that pairs of parabolic subalgebras q, q H which are constructed this way are well aligned. For a unitary character λ of of L we define following Vogan/Zuckerman the unitary representations A q (λ). In this paper we consider the example of the group G = SL(4, R). There are 2 G-conjugacy ( classes ) of skew ( symmetric ) matrices( with representants Q = and Q ) J 0 J 0 0 0 J 2 = where J =. 0 J 0 Let H respectively H 2 be the symplectic subgroups defined by these matrices and H, H 2 the centralizer of Q, respectively Q 2. All these subgroups are fixpoint sets of involutions σ i, i =, 2 and σ i, i =, 2 respectively.

BRANCHING LAWS 3 The matrix Q 2 has finite order, is contained in all subgroups H i and hence defines a θ stable parabolic subalgebra q of sl(4,c) and also θ stable parabolic subalgebras q h = q h of h, respectively q h2 = q h 2 of h 2. Its centralizer L in SL(4, R) is isomorphic to GL(2, C) and the parabolic subgroups q, q h as well as q, q h2 are well aligned. We consider in this paper the unitary representation A q of G corresponding to trivial character λ. Its infinitesimal character is the same as that of the trivial representation. The representation A q was studied from an analytic point of view by S. Sahi [5]. Since the A q has nontrivial (g, K) cohomology and is isomorphic to a representation in the residual spectrum, this representation is also interesting from the point of view of automorphic forms. See for example [7]. We determine in this paper the restriction of A q to the 4 subgroups H i and H i, (i =, 2). After introducing all the notation in section we prove in section 2 using a result of T. Kobayashi, that the restriction of A q to H and H is a direct sum of irreducible unitary representations, whereas the restriction to H 2 and H 2 is a direct integral and doesn t have any discrete spectrum. This discrete/continuous alternative, see [9] is one of the deep results that we invoke for symmetric subgroups. In section 3 and 4 we determine the representations of H respectively of H that appear in the restriction of A q to H respectively H and show that it is a direct sum of unitary representations of the form A q h (µ) respectively A q h (µ ), each appearing with multiplicity one. The main point is here, that we find a natural model in which to do the branching law, based on the existence results of T. Kobayashi; and also following experience from some of his examples, where indeed derived functor modules decompose as derived functor modules (for the smaller group). In section 5 we formulate a conjecture about the multiplicity of representations in the restriction of representations A q of semisimple Lie groups G to subgroups H, which are centralizers of involutions. If the restriction of A q to H is a direct sum of irreducible representation of H we expect that there is a θ-stable parabolic subalgebra q H of H so that all representations which appear in the restriction are of the form A q H(µ) and that a Blattner-type formula holds. See the precise conjecture at the end of section 5, where we introduce a natural generalization of previously known Blattner-type formulas for the maximal compact subgroup. In section 6 we these results are used to construct automorphic representations of Sp(2, R) and GL(2, C) which are in the discrete spectrum for some congruence subgroup. For Sp(2, R) these representations are

4 BENT ØRSTED AND BIRGIT SPEH in the residual spectrum, whereas for GL(2, C) these representations are in the cuspidal spectrum. We expect that our methods extend to other situations with similar applications to automorphic representations; and we hope the point of view introduced here will help to understand in a more explicit way the branching laws for semisimple Lie groups with respect to reductive subgroups. We would like to than T. Kobayashi for helpful discussions during a visit of the second author at RIMS and for the suggestion to also include the restriction to GL(2, C). The second author would also like to thank the University of Southern Denmark in Odense for its hospitality during which part of the research was completed. I. Notation and generalities I. Let G be a connected linear semisimple Lie group. We fix a maximal compact subgroup K and Cartan involution θ. Let H be a θ stable connected semisimple subgroup with maximal compact subgroup K H = K H. We pick a fundamental Cartan subalgebra C H = T H A H of H. It is contained in a fundamental Cartan subalgebra C = T A of G so that T H = T H and A H = A H. The complex Lie algebra of a Lie group (as before) is denoted by small letters and its real Lie algebra by a subscript o. We denote the Cartan decomposition by g o = k o p. Definition: Let q and q H be θ-stable parabolic subalgebras of g, respectively h. We say that they are well aligned if q H = q h We fix x o in T H. It defines well aligned θ-stable parabolic subalgebras q = l u and q H = l H u H = q h of g respectively h; for details see page 274 in [7]. We write L and L H for the centralizer of x 0 in G and in H respectively. For a unitary character λ of L we write λ H for the restriction of λ to L ˆH. I.2 For later reference we recall the construction of the representations A q (V ), V an irreducible (q, L K) module. We follow conventions of the book by Vogan/Knapp [7] (where much more detail on these derived functor modules is to be found) and will always consider representations of L and not of the metaplectic cover of L as some other authors. We consider U(g) as right U(q) module and write V = V top u. Let p L

BRANCHING LAWS 5 be a L K -invariant complement of l k in l. We write r G = p L u. Consider the complex 0 Hom L K (U(g), Hom( 0 r G, V )) K Hom L K (U(g), Hom( r G, V )) K Hom L K (U(g), Hom( 2 r G, V )) K... Here the subscript K denotes the subspace of K-finite vectors. We denote by T (x, U( )) an element in Hom L K (U(g), Hom C ( n r G, V )) K. The differential d is defined by d T (x, U(X X 2 X n )) n = ( ) i T (X i x, U(X X 2 ˆX i X n )) + i= n ( ) i+ T (x, X i U(X X 2 ˆX i X n )) i= + ( ) i+j T (x, U(P rg [X i, X j ] X X 2 ˆX i ˆX j X n )) i<j where x U(g), X j r G and P rg is the projection onto r G. Let s = dim(u k) and let χ be the infinitesimal character of V. If 2 < χ + ρ(u), α > α 2 {0,, 2, 3... } for α (u) then the cohomology is zero except in degree s and if V is irreducible this defines an irreducible ((U(g), K) module A q (V ) in degree s (8.28 in [7]). By (5.23 [7]) the infinitesimal character of A q (V ) is χ + ρ G. If V is trivial the infinitesimal character of A q (V ) is trivial and we write simply A q. Two representations A q and A q are equivalent if q and q are conjugate under the compact Weyl group W K. For an irreducible finite dimensional (q H, L H K) module V LH we define similarly the (U(h), K H ) modules A q H(V LH ). I.3 Let H be the fixpoint set of an involutive automorphism σ of G. We write g o = h o s o for the induced decomposition of the Lie algebra. T. Kobayashi proved [9] that the restriction of A q to H decomposes as direct sum of irreducible representations of H iff A q is K H -admissible, i.e if every K H -type has finite multiplicity. If A q is discretely decomposable as an (h o, K H) module we call an irreducible (h o, H K) module π H an H type of A q if Hom (ho,k H )(π H, A q ) 0

6 BENT ØRSTED AND BIRGIT SPEH and the dimension of Hom (ho,k H )(π H, A q ) its multiplicity. We have l = l H l s. Put u H = u h. The representation of l H on u is reducible and as l H module u = u H (u s). Let q = l u be the opposite parabolic subgroup. Then h = k h u H u H and (u s) u s is a l H -module. As an l H -module g = h (u s) (l s) (u s). I.4 Now let G = SL(4, R). The skew symmetric matrices ( ) J 0 Q = 0 J and ( ) J 0 Q 2 = 0 J ( ) 0 with J = represent the conjugacy classes of skew symmetric matrices. under G. They define symplectic forms also denoted by 0 Q and Q 2. Let H, respectively H 2, be the θ stable symplectic subgroups defined by Q, respectively Q 2. These subgroups are fixpoints of the involutions σ i (g) = Q i (g ) tr Q i, i =, 2. Since Q and Q 2 are conjugate in GL(4, R), but not in SL(4, R), the symplectic groups H and H 2 are not conjugate in SL(4, R). Let H and H 2 be the fixpoints of the involutions σ i(g) = Q i g Q i. Both groups H and H 2 are isomorphic to GL(2, C), but they are not conjugate in SL(4, R). I.5 We fix x 0 = Q 2. It has finite order and is contained in 2 i= H i and in 2 i= H i. It defines a θ stable parabolic subalgebra q of sl(4, C) and also θ stable well aligned parabolic subalgebras q H of the subalgebras h. Its centralizer L = L x0 in SL(4, R), the Levi subgroup, is isomorphic to GL(2, C) = H. For a precise description of the parabolic see page 586 in [7] Let A q be the representation holomorphically induced from q which has a trivial infinitesimal character. This representations is a subrepresentation of a degenerate series representation induced from a one dimensional representation of the parabolic subgroup with Levi factor S(GL(2, R) GL(2, R)) and thus all its K types have multiplicity one. See [5] for details.

BRANCHING LAWS 7 The next proposition demonstrates how different imbeddings of the same subgroup (symplectic res. general linear complex) gives radically different branching laws; in essence it is contained, as a qualitative result, in the criteria of T. Kobayashi, see [9]. Proposition I.. () The restriction of A q to H and to H is a direct sum of irreducible representation each appearing with finite multiplicity. (2) The restriction of A q to H 2 and to H 2 is a direct integral. Proof: Since K H = K H and K H 2 = K H 2 it suffices by T. Kobayashi s theorem (characterizing admissibility, and the theorem of the alternative discrete/continuous in the present situation) to show that A q is K H admissible but not K H 2 -admissible. This will be done in the next section. II. The restriction of A q to K H i, i =, 2. We use in this section the notation introduced on page 586-588 in Knapp/Vogan [7]. ( ) 0 θj II. The Cartan algebra t o of so(4, R) consists of 2-by 2 blocks θ j 0 down the diagonal. We have a θ stable( Cartan subalgebra ) h o = t o a o xj 0 where a o consists of the 2-by 2 blocks. We define e 0 x j h j by ( ) xj iy e j j = y iy j x j j and f j h by Then the roots (u) of (h, u) are ( ) xj iy f j j = x iy j x j. j e + e 2 + (f f 2 ), e + e 2 (f f 2 ), 2e, 2e 2 and a compatible set of positive roots + (l) of (h, l) are e e 2 + (f f 2 ), e e 2 (f f 2 ) The the roots α = e + e 2, α 2 = e e 2 are compatible positive roots of the Lie algebra k with respect to t.

8 BENT ØRSTED AND BIRGIT SPEH The highest weight of the minimal K type of A q is Λ = 3(e + e 2 ). See page 588 in [7]. All other K types are of the form Λ + m (e + e 2 ) + 2m 2 e, m, m 2 N. α α2 Figure The positive root of K H 2 = K L is α 2 and so the restriction of each K type with highest weight (m + 3)α is sum of characters d α with (m + 3) d (m + 3). Hence A q is not K H 2 finite. The groups H and H 2 are conjugate under the outer automorphism which changes the sign of e 2. Hence the simple positive root of K H can be identified with α. A K-type with highest weight Λ + m (e + e 2 )+2m 2 e is a tensor product of a representation with highest weight (3 + m + m 2, 3 + m + m 2 ) and a representation with highest weight (m 2, m 2 ). Its restriction to K H is a direct sum of representations with highest weights (3+m +m 2 +i, 3+m +m 2 i), m 2 i m 2. Figure 2 shows the highest weights of the K H -types for the restriction of A q to K H. Their multiplicities are indicated by a number.

BRANCHING LAWS 9 2 3 4 5 6 2 3 4 5 5 2 3 4 4 4 2 3 3 3 3 2 2 2 2 2 α α2 Figure 2 Thus A q is K H -finite. This completes the proof of proposition I.. II.2 A second series of representations is obtained if we define the parabolic subalgebra q 2 using Q. In this case we obtain an irreducible representation A q2. The restriction of A q2 to H is a direct integral and the restriction to H 2 is a direct sum of irreducible unitary representation. The representation A q of GL(4, R) obtained by inducing an representation A q and a trivial character of the positive scalar matrices is irreducible and unitary. Its restriction to SL(4, R) is equal to A q A q2. Hence the restriction of A q to H has discrete and continuous spectrum. T. Kobayashi showed that for a connected group G the restriction of a representation to the fixed point set of an involution either has only discrete spectrum or only continuous spectrum. So this example shows that his theorem does not hold for disconnected groups G.

0 BENT ØRSTED AND BIRGIT SPEH III. The restriction of A q to the symplectic group H In this section we determine H types of A q. Our techniques are based on homological algebra and the construction of an enlarged complex whose cohomology computes the restriction. We introduce it in III. for semisimple connected Lie groups H and connected reductive subgroups H. Then we will compute the restriction to H by restricting A q to a subgroup conjugate to H. The motivation for this enlarged complex or branching complex is the same as when one is restricting holomorphic functions to a complex submanifold, and identifying the functions with their normal derivatives along the submanifold. In our case we are working with (formalizations of) differential forms satisfying a similar differential equation, so it is natural to try to identify them with their normal derivatives ; this is what is formalized in our definition. As it turns out, with the appropriate conditions (well aligned parabolic subgroups, vanishing of the cohomology in many degrees, and the non-vanishing of explicit classes corresponding to small K-types) we can indeed make the calculation of the branching law effective, at least in the examples at hand. III. We define the enlarged complex for semisimple connected Lie groups H and connected reductive subgroups H which are invariant under the Cartan involution. Let C λh = λ top (u s). Then C λ = C λ top u = C λh top u H = C λ H. Consider the enlarged complex (3.) (Hom L K H (U(g), Hom( i r H, C λ H )) K H, d H ). As a left U(l H ) module U(g) = Q U(h) where Q is S(s). (See [7] 2.56.) We have Hom L K H (U(g), Hom( i r H, C λ H )) K H = Hom L K H (Q U(h), Hom( i r H, C λ H )) K H = Hom L K H (U(h), Hom( i r H, Q C λ H ) K H ) K H. U(g) acts on the enlarged complex from the right and a quick check shows that d H also commutes with this action and therefore we have an action of U(g) on the cohomology of the complex. We have r G = r H (u s) (p L s), and so i r G = l+k=i k r H l (u s p L s).

BRANCHING LAWS The restriction map res H : Hom K L (U(g), Hom( i r G, C λ )) K Hom L K H (U(h), Hom( i r H, Q C λ H ) K H ) K H commutes with the right action of U(g) and induces a restriction map res i H on cohomology. III.2 For the rest of the section we assume that G = SL(4, R). We will show that there exists a symplectic subgroup, which we denote by H w conjugate to H by an element w, so that the restriction map res H is not zero. Since the restriction of A w q depends only on the conjugacy class of H this determines the restriction. Since 0 0 0 0 0 0 0 0 0 0 0 0 Q 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 By abuse of notation we will also write H and H for the groups defined by the skew symmetric form 0 0 0 0 0 0 0 0 0. 0 0 0 Thus ( ) A X h = Y A tr for symmetric matrices X and Y. Recall that q is defined by Q 2 = ( J 0 0 J and that g = h s. We need the fine structure of the parabolic relative to the symmetric subgroup, in order to compare the cohomologies during the branching, as in the following lemmas. Lemma III.. Under the above assumptions a.) l o (h ) o is isomorphic to sl(2,r) R and dim u h =3, b.) the representation of L H acts by a nontrivial character µ with differential (e + e 2 ) on the one dimensional space u s, c.) l s is a direct sum of the trivial representation and the adjoint representation of l h. )

2 BENT ØRSTED AND BIRGIT SPEH d.) u k = u k h has dimension. Proof: We have l o (h ) o = a b x 0 b a 0 x y 0 a b 0 y b a. The nilradical of a parabolic subalgebra with this Levi subalgebra has dimension 3. The dimension of l h k is 2. Hence the dimension of u k h is. On the other hand the dimension of l k is 4. So the dimension of u k is. Since u k h u k we have equality. u s is in the roots spaces for roots e + e 2 + (f f 2 ) and e + e 2 (f f 2 ). Hence l h k acts on u s by e + e 2. l h acts on the 4 dimensional space l s via the adjoint representation. The representation of L H on the symmetric algebra S((u s ) (u s ) (l s )) is isomorphic to a direct sum of representations µ n µ m ad r n N, m N, r N. The parameter λ H µ n, 0 n is in the good range [7] and thus the representation on the cohomology in degree = dim (u k h) of the enlarged complex has composition factors isomorphic to A qh (λ H µ n ) where 0 n. In particular A qh (λ H ) is an (h, K H )-submodule module of the cohomology of the enlarged complex in degree. Lemma III.2. The map res i H is injective in degree. Proof: Note that = dim u k = dim u k h is the degree denoted by s in [7]. It is the degree, in which the complexes defining the representations A q and A q h (λ H ) have nontrivial cohomology. Recall the definition of the K-module L K s (λ) from V.5.70 in [7]. We have bottom layer maps of k o modules. L K (λ) A q and L K H (λ H ) A q h (λ H ) where λ 0 is the trivial character of L K. See theorem V.5.80 of [7] The minimal K type, respectively K H -type is in the image of this map. We have the commutative diagram

BRANCHING LAWS 3 A q res H A q h (λ H ) L K (λ) res K H L K H (λ H ) The minimal K-type of A q has highest weight (3e +3e 2 ). Its restriction to H K is irreducible and equal to L K H s (λ H ), the minimal K H type of A q h (λ H ). Thus res s H is nonzero on minimal K- type and since A q is irreducible it is also injective. Theorem III.3. The representation A q restricted to H is the direct sum of the representations A q h (µ n λ H ), n N, each occurring with multiplicity one. Proof: By the proof of the lemma A q h (λ H ) is a submodule of the restriction of A q to the symplectic group H. Its minimal K H type is also a K type of A q, where occurs with multiplicity one. Hence A q h (λ H ) is a H type of A q with multiplicity one. The minimal K H -type of A q h (λ) has highest weight λ + 3e + 3e 2. The roots of u h p are 2e, 2e 2. Applying successively the root vectors to the highest weight vector of the minimal K H type of A q h (λ) we deduce that A q h (λ) contains the K H types with highest weight ((3 + 2r )e + (3 + 2r 2 )e + λ), r, r 2 N. Theorem 8.29 in [7] show that all these K H types have muliplicity one. Figure 3 shows the K H - type multiplicities of A q (λ H ). Note that we are here using quite a bit of a priori information about the derived functor modules for the smaller group; on the other hand, the branching problem has essentially been reduced to one for compact groups, K-type by K-type.

4 BENT ØRSTED AND BIRGIT SPEH α α2 Figure 3 The Borel subalgebra of k h acts on on the one dimensional space u s by a character µ with differential (e + e 2 ). Let Y 0 be in u s and v 0 a highest weight vector of the minimal K-types of A q. Then Y n v 0 is also the highest weight of an K H -type of highest weight (3 + n)e + (3 + n)e 2 of A q. Let X k 0 be in u k. The linear map T : U(g) s r G C λ 0 which maps to X k C λ 0 is non zero in cohomology and its class [T s ] is the highest weight vector of the minimal K-type. But Y [T s ] Hom L K H (s U(h ), Hom( s r H, C λ H )) Hom L K H (U(h ), Hom( i r H, s C λ H ) K H ) K H. Hence [T s ] A q h (µ + λ H ) and thus A q h (λ H + χ O ) is a H -type of A q. The same argument shows that A q h (λ H + nµ o ), n N, is a H -type of A q. Now every K-type with highest weight (n, n) has multiplicity n 2 and is contained in exactly n 2 composition factors. The multiplicity

BRANCHING LAWS 5 computations in section 2 now show that every composition factor is equal to A q h (λ H + nχ O ) for some n. See Figure 4 2 3 4 5 6 2 3 4 5 5 2 3 4 4 4 2 3 3 3 3 2 2 2 2 2 A q h (λ H + 5χ 0 ) A q h (λ H + 4χ 0 ) A q h (λ H + 3χ 0 ) A q h (λ H + 2χ 0 ) A q h (λ H + χ 0 ) A q h (λ H ) α α2 Figure 4 Using prop. 8. in [7] we deduce that Corollary III.4. Let V be and irreducible (h, K H )-module. Then dim Hom h,k H (V, A q ) = dim Hom (l h ),K H L(H (u h, V ), S i (u s ) C H ) i III.3 The maximal abelian split subalgebra a in l o (h ) o are the diagonal matrices. So parabolic subgroup of the Langlands parameter of the H types of A q is the socalled mirabolic with abelian nilradical. The other Langlands parameter can de determined using the algorithm in [7].

6 BENT ØRSTED AND BIRGIT SPEH IV. Restriction of A q to the complex general linear group H In this section we describe H types of A q using the same techniques as in the previous section. III. As for H we consider the enlarged complex (4.2) (Hom L K H (U(g), Hom( i r h, C λ H )) K H, d H ). and the map res H : Hom K L (U(g), Hom( i r G, C λ )) K Hom L K H (U(h ), Hom( i r H, Q C λ H ) K H ) K H We write g = h s. The intersection u s is 2-dimensional and the representation of the group of L H on u s is reducible and thus a sum of 2 one dimensional representations χ χ 2. The weights of these characters are 2e and 2e 2. So S(u s ) is a direct sum of one dimensional representations of L H with weights 2m e + 2m 2 e 2. In the cohomology in degree of the enlarged complex we have composition factors A q h (λ H χ n χ n 2 2 ) with 0 n, n 2. In particular A q h (λ H ) is an (h, K H )-submodule module of the cohomology in degree. Lemma IV.. The map res H : is injective. Proof: The maximal compact subgroups of H and H are identical. Thus dim u k h = dim u k h = and the minimal K type is irreducible under restriction to K H. Now the same argument as in lemma III. completes the proof. Theorem IV.2. The restriction of A q to H is a direct sum of irreducible representations A q h (λ H χ n χ n 2) 2, n, n 2 N. Their minimal K H -types have highest weights (3+m +m 2 +i, 3+m +m 2 i), m 2 i m 2. Each representation occurs with multiplicity one. Proof: As in the previous section where we proved that the representations A q h (λ H χ n χ n 2 2 ), n, n 2 N appear in the restriction of the A q to H The K H -types of all unitary representations of GL(2, C) have multiplicity one. If the minimal K H -type has highest weight l e + l 2 e 2 2, then the highest weights of the other K H -types are (l + j)e + (l 2 + j)e 2. Multiplicity considerations of K H -types of A q conclude the proof.

BRANCHING LAWS 7 Figure 5 shows the decomposition into irreducible representations. The highest weights of the K H types of a composition factors lie on the lines. For each highest weight there is exactly one composition factor which has a K H type with this weight as a minimal K H - type. 2 3 4 5 6 2 3 4 5 5 2 3 4 4 4 2 3 3 3 3 2 2 2 2 2 α α2 Figure 5 Corollary IV.3. Let V be an irreducible (h, (K H )) module. then dim Hom h,k H (V, A q) = dim Hom (l h ),K H L (H (u h, V ), S i (u s ) C H ) i IV.2 All the H -types A q h (λ H χ n χ n 2 2 ) of A q are simply unitarily induced principal series representations of GL(2, C).

8 BENT ØRSTED AND BIRGIT SPEH V. A conjecture V. The examples in the previous section and the calculations in [] support the following conjecture: Let H be the connected fixpoint set of an involution σ. We write again g = h s. Let A q (λ) be a representation, which satisfies Kobayashi s criterion and thus decomposes discretely, when restricted to H. Suppose q, q h are defined by x o T H. Since A q = A p if q and p are conjugate under the compact Weyl group W K we use the following Definition: Let y o T H and let p, p H be well aligned parabolic subalgebras defined by y o. We call the well aligned parabolic subalgebras p, p H related to q, q H, if x o and y o are conjugate by an element in the compact Weyl group W K of K with respect to T. If x o and y o are not conjugate by an element in the Weyl group W K H of (K H, T H ) then the parabolic subalgebras q H, p H of H are not conjugate in H and thus we have up to conjugacy at most W K /W H K different pairs of well aligned pairs of θ - stable invariant parabolic subalgebras which are related to q, q H. If G = SL(4, R), H = H and (q, q h ) is the pair of well aligned parabolic subalgebras defined by x 0 = Q 2, there there are at most 2 related pairs of well aligned parabolic subalgebras. We expect the following Blattner-type formula to hold for the restriction to H: Conjecture There exists a pair p, p H of well aligned θ stable parabolic subgroups related to q, q H so that every H-type V of A q is of the form A p H(µ) for a character µ of LH ˆ and that dim Hom h,k H(V, A p ) = ( ) s j dim Hom L H (H j (u h, V ), S i (u s) C λh ) i j Some of the characters µ in this formula may be out of the fair range as defined in [7], but that nevertheless the examples discussed in [?] show that there may still be unitary representationis so defined. VI. An application to automorphic representations We use here our results to give different construction of some automorphic representations of Sp(2, R) and GL(2, C). We first explain the ideas in VI. in a more general setting. Again we may consider

BRANCHING LAWS 9 restrictions, this time in the obvious way of restricting functions on locally symmetric spaces to subspaces. VI. Assume first that G is a semisimple matrix group and Γ an arithmetic subgroup, H a semisimple subgroup of G. Then Γ H = Γ H is an arithmetic subgroup of H. Let V π L 2 (G/Γ) be an irreducible (g, K)-submodule of L 2 (G/Γ). If f V π then f is a C - function and so we define f H as the restriction of f to H/Γ H. Lemma VI.. The map is an (h, K H) map. Proof: Let h, h o H. Then RES H : V π C (H/Γ H ) f f H ρ(h)f(h o ) = f(h h 0 ) = f H (h h 0 ) = ρ(h)f H (h o ). Suppose that the irreducible unitary (g, K)- module π is a submodule of L 2 (G/Γ) and that its restriction to H is a direct sum of unitary irreducible representations. Proposition VI.2. Under the above assumptions RES H (π) is nonzero and its image is contained in the automorphic functions on H/Γ H. Proof: Let f H be a function in RES H (π). Then by section it is K H finite and we may assume that it is an eigenfunction of the center of U(h). Let g 2 = tr(g g). Since sup g G F (g) g r <, the same is true for f H and so f H is an automorphic function on H/Γ H. The functions in the (g, K)- module π L 2 (G/Γ) are eigenfunctions of the center of the enveloping algebra U(g) and are K-finite, hence analytic. Thus if f π L 2 (G/Γ) then there exists W U(g) so that W f(e) 0. Hence RES H (W f) 0. VI.2 Now we assume that G = GL(4, R) and that Γ GL(4, Z) is a congruence subgroup. The groups Γ = Γ H and Γ = Γ H are arithmetic subgroups of Sp(4, R), respectively GL(2, C). Recall the definition of the (g, K)- module A q from II.. It is a submodule of L 2 (Z\G/Γ) for Γ small enough where Z the connected component of the center of GL(4, R). We will for the remainder of this sections consider it as an automorphic representation in the residual spectrum

20 BENT ØRSTED AND BIRGIT SPEH [7]. Then RES H (A q ) and RES H (A q ) are nonzero. Its discrete summands are contained in the space of automorphic forms. Theorem VI.3. The discrete summands of the representation RES H (A q ) respectively RES H (A q ) are subrepresentations of the discrete spectrum of L 2 (H /Γ H ), respectively L 2 (H /Γ H ). Proof: All the functions in A q decay rapidly at the cusps. Since the cusps of H /Γ H are contained in the cusps of G /Γ this is true for the functions in RES H (A q ). Thus they are also contained in the discrete spectrum. For Sp(2, R) the representations constructed in the previous theorem were first described by H. Kim, see [6]; see also [6]. For GL(2, C) we obtain the stronger result Theorem VI.4. The representations in the discrete spectrum of RES H (A q ) are in the cuspidal spectrum of L 2 (H /Γ H ). Proof: By IV.2 the representations in the discrete spectrum of RES H (A q ) are unitarily induced principal series representations and so by a result of Wallach they are in fact cuspidal representations. The embedding of H = GL(2, C) into SL(4, R) is defined as follows: Write g = A + ib with real matrices A, B. Then g ( A B B A Thus Γ H is isomorphic to a congruence subgroup of GL(2, Z[i]). Since all the representations in the discrete spectrum of the restriction of A q do have nontrivial (h, K h ) cohomology with respect to some irreducible finite dimensional nontrivial representation F we obtained the well known result [2], [4] Corollary VI.5. There exists a congruence subgroup Γ GL(2, Z[i]) and a finite dimensional non-trivial representation F of GL(2, Z[i]) so that H i (Γ, F ) 0 for i =,2 ).

BRANCHING LAWS 2 References [] Gross, B., Wallach, N., Restriction of small Discrete Series representations to symmetric subgroups, in The mathematical legacy of Harish-Chandra, Proc. Sympos. Pure Math., 68, 2000, 255-272. [2] Harder,G. On the cohomology of SL(2,O), Lie groups and their representations, Proceedings Summer school on Groups representations of the Bolyai Janos Math. Soc. Budapest. 97 Halsted, New York 975, [3] Hecht, H., Schmid, W., A proof of Blatner s conjecture, Inv. Math. Vol 3 (976), p. 29-54. [4] Howe, R., Reciprocity laws in the theory of dual pairs, Representation Theory of reductive groups, P. Trombi (editor), Progress in Math. v. 40, (983), 56-76. [5] Jacobsen, H. P., Vergne, M., Restriction and expansions of holomorphic representations, Journ. of Funct. Anal. 34 (979), 29-53. [6] Kim, H., The residual spectrum of Sp 4, Comp. Math 99(995), 29-5. [7] Knapp, Anthony W.; Vogan, David A., Jr. Cohomological induction and unitary representations. Princeton Mathematical Series, 45. Princeton University Press, Princeton, NJ, 995. xx+948 pp. ISBN: 0-69-03756-6 [8] Kobayashi, T., Harmonic Analysis on Homogeneous Manifolds of Reductive Type and Unitary Representation Theory, in Selected Papers on harmonic Analysis, groups and invariants, K. Nomizu (Editor), Amer. Math. Soc. Transl. Ser. 2, v. 83. -33, 999, and references therein. [9] Kobayashi, T., Discretely decomposable restrictions of unitary representations of reductive Lie groups - examples and conjectures, in Analysis on Homogeneous Spaces and Representation Theory of Lie Groups, T. Kobayashi (Editor), Advanced Studies in Pure Mathematics v. 26, 99-27, Kinokuniya, Tokyo, 2000. [0] Kobayashi, T., Discrete Series Representations for the Orbit Spaces Arising from Two Involutions of Real Reductive Groups, Journ. of Funct. Anal. 52 (998) p. 00-35. [] Kobayashi, T., Discrete decomposability of the restriction of A q (λ) with respect to reductive subgroups III, Inv. Math. 3 (997), 229-256. [2] Loke, H., Restrictions of quaternionic representations, Journ. of Funct. Anal. 72 (2000), 377-403. [3] Martens, S., The characters of the holomorphic Discrete Series, Proc. Nat. Acad. Sci. USA Vol. 72,(976), 3275-3276. [4] Rohlfs, J. On the cuspidal cohomology of the Bianchi Modular Groups, Math. Z. 88 (985), 253-269. [5] Sahi, S. and Stein, E., Analysis in matrix space and Speh s representations. Invent. Math. 0 (990), no. 2, 379 393. [6] Schwermer, J. On Euler products and residual cohomology classes for Siegel modular varieties, Forum Math. 7 (995), -28 [7] Speh, B. Unitary representations of GL(n, R) with nontrivial (g, K) cohomology, Invent. Math. 7 (983) 443 465. [8] Vargas, J., Restriction of some Discrete Series Representations, Algebras, Groups and Geometries, Vol. 8, (200), p. 85-00. [9] Vargas, J., Restriction of holomorphic Discrete Series to real forms. To appear, Rendiconti di Torino.

22 BENT ØRSTED AND BIRGIT SPEH [20] Wolf, J. Representations that remain irreducible on parabolic subgroups. Differential Geometrical Methods in Mathematical Physics IV (Proceedings, Aix en Provènce and Salamanca, 979), Springer Lecture Notes in Mathematics, vol. 836 (980), pp. 29 44. [2] Zhang, G., Berezin transform of holomorphic Discrete Series on real bounded symmetric domains. Trans. Amer. Math. Soc., Vol. 353, (200), 3769-3787. Bent Ørsted, Department of Mathematics, University of Aarhus,, Aarhus, Denmark E-mail address: orsted@imf.au.dk Birgit Speh, Department of Mathematics, 30 Malott Hall, Cornell University, Ithaca, NY 4853-420, U.S.A. E-mail address: speh@math.cornell.edu