SOME REMARKS ON DEGENERATE PRINCIPAL SERIES. Chris Jantzen

pacific journal of mathematics Vol. 186, No. 1, 1998 SOME REMARKS ON DEGENERATE PRINCIPAL SERIES Chris Jantzen In this paper, we give a criterion for the irreducibility of certain induced representations, including, but not limited to, degenerate principal series. More precisely, suppose G is the F -rational points of a split, connected, reductive group over F, with F = R or p-adic. Fix a minimal parabolic subgroup P min = AU G, with A a split torus and U unipotent. Suppose M is the Levi factor of a parabolic subgroup P P min, and ρ an irreducible representation of M. Further, we assume that ρ has Langlands data (A, λ) in the subrepresentation setting of the Langlands classification (so that ρ Ind M P min M (λ 1)). The criterion gives the irreducibility of Ind G P (ρ 1) if a collection of induced representations, induced up to Levi factors of standard parabolics, are all irreducible. This lowers the rank of the problem; in many cases, to one. The approach used to obtain our criterion is based on an argument in [Tad1]. We note that, since F may be real or p-adic, this gives an instance of Harish-Chandra s Lefschetz principle in action (cf. [H-C]). We now discuss the contents, section by section. In the first section, we review notation and give some background. The second section contains the main irreducibility result (cf. Theorem.6). This tells us the induced representation is irreducible if three conditions denoted ( ), ( ), ( ) all hold. As mentioned above, these conditions involve the irreducibility of representations induced up to Levi factors of standard parabolic subgroups. In the third section, we show that ( ) implies ( ) (cf. Proposition 3.3). In the fourth section, we give a more explicit description of ( ) when G = Sp n (F) (cf. Lemma 4.1), and use this to show that ( ) implies ( ) for G = Sp n (F) (cf. Proposition 4.). To show that ( ) implies ( ) for G = Sp n (F), we use the results of [Gol], so that, at this point, we assume that if F is p-adic, charf = 0. We also discuss other cases where ( ) implies ( ), and give an example where it does not. In the fifth section, we apply these results to the example of degenerate principal series for Sp n (F), i.e., P = MU is maximal and ρ is one-dimensional (cf. Corollary 5.). We also compare the results to known results for degenerate principal series for Sp n (F), to 67

68 CHRIS JANTZEN see that the converse to Theorem.6 fails. Finally, in the sixth section, we do two more examples. One is an application of our results to degenerate principal series for G (F ); the other is an example of how some of the ideas used here can be applied to show irreducibility for a degenerate principal series representation of Sp n (F) not covered by Corollary 5.. Comments and questions from Dihua Jiang, Henry Kim, Soo Teck Lee, and the referee helped in the preparation of this paper. My thanks go out to all of them. 1. Notation and preliminaries. In this section, we introduce notation and give some background results. Note that this paper only deals with admissible representations; in what follows, any representation introduced is assumed to be admissible. Throughout this paper, F = R or F p-adic. In the fourth and fifth sections, we also assume that if F is p-adic, charf = 0. We let denote absolute value on F ; for Fp-adic, it is normalized so that ϖ = q 1, where ϖ is a uniformizer and q is the order of the residue field. In this paper, G denotes the F-points of a split, connected, reductive group over F. Let P min denote a minimal parabolic subgroup for G. It admits a Levi decomposition P min = AU, with A a split torus and U unipotent. We let denote the set of roots of G with respect to A; nd the subset of nondivisible roots. The choice of P min determines a subset + of positive roots. Let Π + denote the simple roots. The Weyl group is W = N G (A)/A. If s α denotes the reflection associated to α, then we may also characterize the Weyl group as W = s α α Π. Let l denote the length function on W, i.e., l(w) is the number of simple reflections in a reduced expression for w. We let w 0 denote the longest element of W (note that w0 = 1). Fix P min. The standard parabolic subgroups, i.e., those containing P min, are parameterized by subsets Φ Π. In particular, P Φ = P min, {s α } α Φ }. (We will not be too careful about distinguishing between an element of W and a representative for it in G.) All parabolic subgroups are conjugate to a standard parabolic subgroup. If P is a standard parabolic subgroup with Levi factorization P = MU, we call M a standard Levi subgroup. If M is the standard Levi subgroup associated to Φ M Π, we let M denote the roots for M, + M = M + the positive roots for M; the simple roots for M are Φ M. Set W M = N M (A)/A = s α α ΦM. The following subsets of W will play an important role in this paper: If M, N are standard Levi

DEGENERATE PRINCIPAL SERIES 69 subgroups, set W MN = {w W w( + M) +,w 1 ( + N) + }. W MN may also be characterized as follows: w W has w W MN if and only if w is the minimal length element in the double-coset W N ww M. Let P = MU be a standard parabolic, ρ a representation of M. We let i GM (ρ) = Ind G P (ρ 1) denote the induced representation (normalized induction). It will occasionally be useful to write the inducing representation differently. Let P = MU be a standard parabolic, A M the maximal F -split torus lying in the center of M. Let a M denote the real Lie algebra of A M.ForF p-adic, it is defined by a M = Hom(X(M) F, R), where X(M) F denotes the F -rational characters of M. Then, a M = X(M) F Z R. When M = A, we just write a, a. Note that a M a, a M a. Further, if (, ) denotes the usual W -invariant inner product on a, then a = a M sp R {Φ M }, an orthogonal direct sum. For ν a, we write ν =(ν) M +(ν) M for the corresponding decomposition. There is a homomorphism H PM : M a M defined by e χ,h P M (m) = χ(m) for all m M, χ X(M) F. If ρ is an irreducible essentially tempered representation of M, there is a unique irreducible tempered representation σ of M and unique ν a M such that ρ = σ e ν,h P M ( ). It is in this context that we discuss the Langlands classification. We now turn our attention to the Langlands classification ([Lan], [Sil], [B-W]). Our discussion here most closely resembles that in [Sil]. In this paper, it will be convenient to use the subrepresentation form of the Langlands classification, though we also describe the quotient form below. If P = MU is a standard parabolic, let Π(P, A M ) a M denote the set of simple roots for (P, A M ) and (a M) = {β a M (β,α) < 0 for all α Π(P, A M )}.

70 CHRIS JANTZEN We note that Π(P, A M ) may be viewed as the nonzero projections to a M of Π. If σ is an irreducible tempered representation of M and ν (a M), then i GM (σ e ν,h P ( ) M ) has a unique irreducible subrepresentation (Langlands subrepresentation). Further, for any irreducible (admissible) representation π of G, there is a unique standard P = MU, with a unique (up to equivalence) irreducible tempered representation σ of M and a unique ν (a M), such that π is the unique irreducible subrepresentation of i GM (σ e ν,h P ( ) M ). If ρ = σ e ν,h P ( ) M, we write L(ρ) for the Langlands subrepresentation. Note that L(ρ) occurs with multiplicity one in i GM (ρ). We call (M,ρ) a set of Langlands data. The quotient version of the Langlands classification is similar: If P = MU is a standard parabolic, σ an irreducible tempered representation of M, and ν (a M) + = (a M), then i GM (σ e ν,h P ( ) M ) has a unique irreducible quotient (Langlands quotient). The admissible dual may be classified this way as well. The relationship between the Langlands data for the subrepresentation and quotient versions of the Langlands classification is included in the lemma below. (We note that if one introduces nonstandard parabolic subgroups into the picture, the relationship is immediate from the [B-W] proof: If π has Langlands quotient data (P, ρ), then it has subrepresentation data (P,ρ), and vice-versa, where P denotes the parabolic subgroup opposite P.) Lemma 1.1. Suppose M is a standard Levi subgroup of G and ρ = σ e ν,h P ( ) M has σ irreducible tempered and ν (a M). Then, (1) L(ρ) has Langlands quotient data (M,w 0 ρ), where w 0 W MA of maximal length and M = w 0Mw 0 1 = w 0 Mw 0 (also a standard Levi subgroup). () If denotes contragredient, L(ρ) =L(w 0 ρ). Proof. The proof for G = Sp n (F) is in Section 6 of [Tad1]. In general, for (1), consider the unnormalized standard intertwining operator A P,P (w 0 ρ) : Ind G P ((w 0 ρ) 1) Ind G P ((w 0 ρ) 1) where P denotes the parabolic subgroup opposite P. It is a straightforward matter to check that Ind G P ((w 0 ρ) 1) = i GM (ρ). L(ρ) may be identified as i GM (w 0 ρ)/ ker A P,P (w 0 ρ) or imagea P,P (w 0 ρ). By definition, the former gives L(ρ) as a Langlands quotient; the latter as Langlands subrepresentation.

DEGENERATE PRINCIPAL SERIES 71 For (), first observe that by contragredience, L(ρ) is the unique irreducible quotient of igm (ρ) = i GM ( ρ). Now, ρ = σ e ν,h P ( ) M. Since σ is irreducible tempered and ν (a M) +,(M, ρ) meets the requirement for Langlands quotient data. In particular, it is the Langlands quotient data for L(ρ). Next, note that we can write w 0 = w 0 w 0, where w 0 W M of maximal length. We then observe that (w 0) 1 W M A of maximal length (a consequence of the fact that w 0 w 0 w 0 W M of maximal length). Finally, using (1) to convert back to Langlands subrepresentation data, we see that as claimed. L(ρ) =L(w 0 ρ), Later in this paper, we look at the the example of G = Sp n (F). We now introduce some notation which will be needed then. First, if G = GL n (F ), a standard Levi subgroup has the form M = GL m1 (F ) GL mk (F ), with m 1 + +m k = n. Asin[B-Z], if ρ i, i =1,...,k, is a representation of GL mi (F ), let ρ 1 ρ k = i GM (ρ 1 ρ k ). Next, let J bean n matrix with j a,b =0ifa+b n+1, j a,b = 1 ifa+b=n+ 1 and a n, j a,b =1ifa+b=n+ 1 and a n + 1. Then, Sp n (F)={X GL n (F ) T XJ X = J }. A standard Levi subgroup of G = Sp n (F) has the form M = GL m1 (F ) GL mk (F ) Sp m (F), with m 1 + +m k +m=n. We extend the notation of [B-Z] asin[tad1]: if ρ i, i =1,...,k, is a representation of GL mi (F ) and τ a representation of Sp m (F), we set ρ 1 ρ k τ =i GM (ρ 1 ρ k τ) (using τ = 1 to denote the trivial representation of Sp 0 (F), if needed).. Irreducibility criterion. The main result in this section, Theorem.6, gives irreducibility for certain induced representations if the conditions ( ), ( ), ( ) below hold. In subsequent sections, we show ( ) implies ( ) and, in certain cases, ( ) as well. Suppose that M is a standard Levi subgroup of G, λ a character of A, and (A, λ) satisfies the requirements to be Langlands (subrepresentation) data for M. That is, if we write λ = λ 0 e ν λ,h Pmin ( ), then ν λ (a ) (Φ M ), where Φ M Π is the subset of simple roots corresponding to M and (a ) (Φ M )={β a (β,α) < 0 for all α Φ M }. In particular, there is an irreducible admissible representation of M having Langlands data (A, λ); we denote this representation by L M (λ). Theorem.6 is concerned with showing irreducibility for π = i GM (L M (λ)).

7 CHRIS JANTZEN Definition.1. If M, N are standard Levi subgroups, set W MN A = {w W MN w( M ) N = }, with W MN as in Section 1. If Π = {α 1,...,α n }, let M i denote the standard Levi subgroup associated to {α i } Π. Then, as the first of our three conditions, we suppose that ( ) i MiA(w λ) is irreducible for all i and all w W MMi A. We next give a pair of lemmas which lead up to Proposition.4. Lemma.. Let w W MA with l(w) 1. Then, there is a simple reflection s i and a w W MA such that w = s i w with l(w) =l(w )+1. Proof. Let w = s i1 s ik be a reduced expression for w. We claim that w = s i s ik works. We must check that w W MA. It is easy to check that if w is the shortest element in w W M, w is the shortest element in w W M. Thus, w W MA. Lemma.3. Let M be a standard Levi subgroup, α i Π, and s i the corresponding simple reflection. For w W, we have w W MMi A if and only if w, s i w W MA with l(s i w)=l(w)+1. Proof. First, suppose w, s i w W MA with l(s i w) = l(w) + 1. Observe that since W Mi ww M =(ww M ) (s i ww M ) and l(w) <l(s i w), we have w W MMi.Ifw W MMi A, we must have w( M ) Mi = Mi contains α i, implying s i ww M w 1. However, this forces ww M = s i ww M, contrary to the assumption that w, s i w are both in W MA. Thus, w W MMi A. In the other direction, suppose w W MMi A. Since w W MMi, we already know w W MA and l(s i w)=l(w) + 1. It remains to show s i w W MA. First, we claim that (s i ww M ) (ww M )=. Since w W MMi A, we have w( M ) Mi =, so that s i ww M w 1. The claim follows. To show s i w W MA, we need to check that s i w is the shortest element of s i ww M. Suppose not say w s i ww M and l(w ) <l(s i w). Then, W Mi ww M = W Mi w W M with l(w ) l(w). Since w W MMi requires w to be the minimal length element of the double coset, this forces w = w. Therefore, w s i ww M, contradicting the observation that s i ww M w 1. Thus, s i w must be the shortest element of s i ww M, finishing up the proof that s i w W MA, and the lemma. Proposition.4. If ( ) holds, then π i GA (λ) = i GA (w λ)

DEGENERATE PRINCIPAL SERIES 73 for all w W MA. Proof. Clearly, π i GA (λ). We show that i GA (λ) = i GA (w λ) by induction on l(w). For l(w) = 0, it is trivial. Suppose w W MA with l(w) > 0. Choose s i, w as in Lemma. above. By the inductive hypothesis, we may assume i GA (λ) = i GA (w λ). Therefore, it suffices to show that i GA (s i w λ) = i GA (w λ). However, by Lemma.3 above, w W MMi A. Therefore, ( ) implies i MiA(w λ) is irreducible. This forces i MiA(w λ) = i MiA(s i w λ). By induction in stages, we then get i GA (w λ) = i GA (s i w λ), as needed. We need the following lemma to set up the second condition for irreducibility. Lemma.5. such that has w 1 ν λ (a ). With λ = λ 0 e ν λ,h Pmin ( ) as above, there is a w 1 W MA λ 1 = w 1 λ = w 1 λ 0 e w1 ν λ,h Pmin ( ) Proof. Since W acts transitively on the Weyl chambers, there is a w 1 W (possibly more than one) such that w 1 ν λ (a ). We claim that such a w 1 necessarily lies in W MA. Fix such a w 1. Then, we have (w 1 ν λ,α) 0 for all α +. Further, since ν λ (a ) (Φ M ), we have (w 1 ν λ,w 1 α)=(ν λ,α)<0 for all α + M. To show w 1 W MA, we must check that w 1 α + for all α + M. Suppose not say α 0 + M with w 1 α 0. If we write w 1 α 0 = α 0, α 0 +, then (w 1 ν λ,w 1 α 0 )=(w 1 ν λ, α 0)= (w 1 ν λ,α 0) 0, contradicting (w 1 ν λ,w 1 α 0 )=(ν λ,α 0 )<0. Thus, we see that w 1 + M +, hence w 1 W MA, as needed. We now give the second irreducibility condition. Fix w 1, λ 1 as in Lemma.5 and set Φ 1 = {α i Π (w 1 ν λ,α i )=0}. Assume that ( ) i MΦ1 A(λ 1 ) is irreducible.

74 CHRIS JANTZEN Observe that if ν λ1 = w 1 ν λ, then (in the notation of Section 1) (ν λ1 ) MΦ1 =0 and (ν λ1 ) MΦ 1 = νλ1. Thus, i MΦ1 A(w 1 λ 0 e ν λ 1,H Pmin ( ) )=(i MΦ1 A(w 1 λ 0 1)) (e ν λ 1,H PΦ1 ( ) ). Since (ν λ1,α)<0 for all α Π\Φ 1, we see that ν λ1 (a M Φ1 ). In particular, (M Φ1,i MΦ1 (λ 1 )) is a set of Langlands data for G. We set π 1 = L(i MΦ1 A(λ 1 )). We now give the third irreducibility condition. Let w 0 W M of maximal length. Also, let λ = λ 1 denote the contragredient of λ. We assume ( ) i MiA(ww 0 λ) is irreducible for all i and all w W MMi A. Theorem.6. Suppose M is a standard Levi subgroup, (A, λ) Langlands data for M, and π = i GM (L M (λ)). If ( ), ( ), ( ) all hold, then π is irreducible. Proof. First, from Proposition.4 and ( ), π i GA (λ) = i GA (w λ) for all w W MA. In particular, π i GA (λ 1 ). From ( ) and the fact that (M Φ1,i MΦ1 A(λ 1 )) is Langlands data for G, we have that π 1 = L(i MΦ1 A(λ 1 )) is the unique irreducible subrepresentation of i GA (λ 1 ) = i GMΦ1 (i MΦ1 A(λ 1 )). Therefore, π 1 must be the unique irreducible subrepresentation of π. Consequently, π 1 is the unique irreducible quotient of π. Next, observe that by Lemma 1.1, L M (λ) =L M (w 0 λ). Therefore, π = i GM ( L M (λ)) i GA (w 0 λ). By Proposition.4 and ( ), we have π i GA (w 0 λ) = i GA (ww 0 λ) for all w W MA. Further, we claim that for a suitable w W MA, we have ww 0 = w 0 w 1. This amounts to verifying that w 0 w 1 w 0 W MA. To see this, take α + M. Then, w 0 α = α for some α + M. Now, since w 1 W MA, w 1 w 0 α = w 1 ( α )= α for some α +. Then, w 0 w 1 w 0 α = w 0 ( α ) +, as needed. For this choice of w, wehave π i GA (ww 0 λ) = i GA (w 0 w 1 λ) = i GA (w 0 λ 1 ).

DEGENERATE PRINCIPAL SERIES 75 Next, we claim that π 1 is the unique irreducible subrepresentation of i GA (w 0 λ 1 ). By Lemma 1.1, if w Φ 1 W MΦ 1 A of maximal length and w Φ 1 W MΦ1 of maximal length, ( ( )) π 1 = L w Φ 1 i MΦ1 A(λ 1 ) ( ( ))) = L w Φ 1 i MΦ1 A ( λ1 ( ( = L w Φ 1 i MΦ1 A (w Φ1 λ ))) 1, where i MΦ1 A( λ 1 ) = i MΦ1 A(w Φ 1 λ 1 )by( ). In particular, by Lemma.7 (below), π 1 i GM Φ1 (w Φ 1 i MΦ1 A (w Φ1 λ )) 1 = iga (w 0 λ ) 1, as unique irreducible subrepresentation, as claimed. Finally, since π 1 i GA (w 0 λ 1 ) as unique irreducible subrepresentation and π i GA (w 0 λ 1 ), we see that π 1 is the unique irreducible subrepresentation of π. On the other hand, we showed above that π 1 is the unique irreducible quotient of π. This means π 1 must appear as both the unique irreducible subrepresentation and unique irreducible quotient of π. However, since π i GA (λ 1 ) = i GMΦ1 (i MΦ1 A(λ 1 )), and π 1 appears with multiplicity one in i GMΦ1 (i MΦ1 A(λ 1 )), π 1 appears with multiplicity one in π. The only way this can happen is if π = π 1, giving the desired irreducibility. Lemma.7. Let L be a standard Levi subgroup, w 0 W LA of maximal length. Set L = w 0 Lw 0 = w 0Lw 0 1 (also a standard Levi subgroup). If χ is a character of A, then w 0 i LA (χ) = i L A(w 0 χ). Proof. Let π 1 = i LA (χ), π = i L A(w 0 χ). Consider the isomorphism c : L L given by c(x) =w 0xw 0 1. One can check that this gives rise to a bijective map C : V π1 V π, where Cf is defined by (Cf)(c(x)) = f(x). One can then verify that C(w 0 π 1 )=π C, giving the equivalence claimed.

76 CHRIS JANTZEN 3. The condition ( ). In this section, we show that the condition ( ) in the previous section implies ( ). Lemma 3.1. Let M be a standard Levi subgroup and w 0 W M of maximal length. Let s i be a simple reflection and M i the standard Levi subgroup for P min,s i. Set M i = w 0 M i w 0. Then, w W MMi A s iw 0 ww 0 W MM i A, where s i = w 0 s i w 0, the simple reflection in M i. Proof. First, we show s iw 0 ww 0 W MA. To do this, we start by checking that α i w 0 ww 0 + M, where α i = w 0 α i Π. Suppose this were not the case. Then, w 0 w 1 w 0 α i + M w 1 w 0 α i M w 1 ( α i ) M α i w + M, contradicting w W MMi A. Thus, α i w 0 ww 0 + M. Now, we check that s iw 0 ww 0 W MA : s iw 0 ww 0 + M = s iw 0 w M s iw 0 = s i +. As long as α i w 0 ww 0 + M and we just checked this we have s iw 0 ww 0 + M +, as needed. Next, we show that s iw 0 ww 0 W AM i. We start by noting that w 0 ( + \ + M)= + \ + M. (This follows easily from the fact that w 0 + M = M and l(w 0 )= nd + M.) Calculate: (s iw 0 ww 0 ) 1 α i = w 0 w 1 w 0 s i α i = w 0 w 1 w 0 ( α i) = w 0 w 1 α i.

DEGENERATE PRINCIPAL SERIES 77 Since w W MMi A, w 1 α i M. Thus, (s iw 0 ww 0 ) 1 α i = w 0 w 1 α i w 0 ( + \ + M) = + \ + M, giving s iw 0 ww 0 W AM i. We now have s iw 0 ww 0 W MA W AM i = W MM i. The final step is to show (s iw 0 ww 0 M ) M i =, or equivalently, w 0 w 1 w 0 s i α i M. However, from above, we have which finishes the proof. w 0 w 1 w 0 s i α i + \ + M, Corollary 3.. The map w s iw 0 ww 0 gives a bijection between W MMi A and W MM i A. Proof. Switching the roles of M i and M i, one gets a map sending w W MM i A to s i w 0 ww 0 W MMi A. This is the inverse map. We now come to the proposition that ( ) implies ( ). Proposition 3.3. Suppose ( ) holds. Then, i MiA(ww 0 λ) is irreducible for all i and all w W MMi A. Proof. By Lemma.7, noting that s iw 0 = w 0 s i is the element of W MiA A maximal length, we have of (s iw 0 ) imia(w λ) = (s iw 0 ) i MiA(w λ) = i M i A(s iw 0 ww 0 (w 0 λ)). Therefore, i MiA(w λ) irreducible implies i M i A(s iw 0 ww 0 (w 0 λ)) irreducible. The proposition then follows from the preceding corollary. 4. The condition ( ). In this section, we show that the condition ( ) of Section implies ( ) when G = Sp n (F). We also show that ( ) implies ( ) for general G under certain conditions on λ. We close by giving an example to show that ( )

78 CHRIS JANTZEN does not imply ( ) in general. We note that the proof that ( ) implies ( ) for G = Sp n (F) uses the results of [Gol]. So, for this section and the next, we assume that if F is p-adic, charf =0. We start by giving a more explicit description of ( ) for Sp n (F). This description will also be useful in the next section. Lemma 4.1. Let G = Sp n (F) and M = GL m1 (F ) GL mk (F ) Sp m (F), a standard Levi subgroup for G, where m 1 + +m k +m = n. Suppose λ =(χ 1,1 χ 1,m1 ) (χ k,1 χ k,mk ) (χ k+1,1 χ k+1,m ). Then, ( ) holds if and only if the following are all irreducible: (1) χ i,j χ i,j with i<i, () χ i,j 1 with i k +1, (3) χ i,j χ 1 i,j with (i) i k +1 and (ii) j >j if i = i. Proof. First, let us discuss W MA λ. We may characterize an element of W MA λ as a shuffle of (1) χ 1,1,χ 1,,...,χ 1,j1 ; (1) χ 1 1,m 1,χ 1 1,m 1 1,...,χ 1 1,j 1+1;.. (k) χ k,1,χ k,,...,χ k,jk ; (k) χ 1 k,m k,χ 1 k,m k 1,...,χ 1 k,j k +1; (k +1) χ k+1,1,χ k+1,,...,χ k+1,m ; for some j 1,...,j k, where 0 j i m i. We use shuffle in the conventional sense: a permutation for which the relative orders within (1), (1),... (k), (k), (k+1) are preserved (e.g., if χ 1 χ n is such a shuffle, and χ r = χ 1 k,m k, χ s = χ 1 k,m k 1, then r<s). Further, any such shuffle, for any j 1,...,j k,is an element of W MA λ. Next, from Lemma.3, we recall that w W MMi A if and only if w, s i w W MA with l(s i w)=l(w) + 1. Suppose i = n. Then, if w W MMn A, write w λ = χ 1 χ n 1 χ r,s (we are really concerned with only the last term). Since s n w W MA and s n w λ = χ 1 χ n 1 χ 1 r,s, we see from the description of W MA λ above that r k. Thus, for w W MMn A, i MnA(w λ) must have the form i MnA(w λ) =χ 1 χ n 1 (χ r,s 1)

DEGENERATE PRINCIPAL SERIES 79 with r k. It is also easy to see from the discussion above that for χ r,s with r k, there is some w W MMn A and some χ 1,...,χ n 1 such that i MnA(w λ) =χ 1 χ n 1 (χ r,s 1). Thus, the case i = n gives rise to condition in the statement of the lemma. Now, suppose i<n. In a similar manner, we can see that for w W MMi A, i MiA(w λ) has one of the following forms: i MiA(w λ) χ 1 χ i 1 (χ r,s χ r,s ) χ i+ χ n, r < r, χ 1 χ i 1 (χ r,s χ 1 r,s ) = χ i+ χ n, r k + 1 and s >sif r = r, or χ 1 χ i 1 (χ 1 r,s χ 1 r,s ) χ i+ χ n, k +1>r>r. Further, for χ r,s,χ r,s with r<r, one can see that for some i, there is a w W MMi A and χ 1,...,χ i 1,χ i+,...,χ n such that i MiA(w λ) =χ 1 χ i 1 (χ r,s χ r s ) χ i+ χ n. This gives rise to condition 1 in the statement of the lemma. Also, for χ r,s,χ r,s with r k + 1 and s >sif r = r, one can see that for some i, there is a w W MMi A and χ 1,...,χ i 1,χ i+,...,χ n such that i MiA(w λ) =χ 1 χ i 1 (χ r,s χ 1 r,s ) χ i+ χ n. This gives rise to condition 3 in the statement of the lemma. Note that χ 1 r,s χ 1 r,s is irreducible if and only if χ r,s χ r,s is irreducible, so the third possibility above adds nothing new to the statement of the lemma. We now give a proposition which says ( ) implies ( ) for Sp n (F). Proposition 4.. Let G = Sp n (F). Suppose λ 1 = w 1 λ, w 1 W MA, and M Φ1 standard Levi subgroup are as in Section. Further, assume ( ) holds. Then, i MΦ1 A(λ 1 ) is irreducible. Proof. Write λ 1 =( α1 ρ 1,1 α1 ρ 1,k1 ) ( α l ρ l,1 α l ρ l,kl ) (ρ l+1,1 ρ l+1,kl+1 ), with α i R, α 1 < <α l <0, and ρ i,j a unitary character of F. Then, i MΦ1 A(λ 1 )=( α1 ρ 1,1 α1 ρ 1,k1 )

80 CHRIS JANTZEN ( α l ρ l,1 α l ρ l,kl ) (ρ l+1,1 ρ l+1,kl+1 1). We observe that ( α1 ρ 1,1 α1 ρ 1,k1 ),...,( α l ρ l,1 α l ρ l,k l ) are all irreducible (more generally, inducing a discrete series representation of a standard Levi subgroup of GL k (F ) always results in an irreducible representation). It remains to use the hypotheses of the proposition (i.e., ( )) to show that (ρ l+1,1 ρ l+1,kl+1 1) is irreducible. Recall that λ is Langlands data for a standard Levi subgroup M = GL m1 (F ) GL mk (F ) Sp m (F). For λ =(χ 1,1 χ 1,m1 ) (χ k,1 χ k,mk ) (χ k+1,1 χ k+1,m ), write χ i,j = βi,j τ i,j with β i,j R and τ i,j a unitary character of F. To be Langlands data for M, wemusthave 1. for i k : β i,1 <β i, < <β i,mi and. for i = k +1:β k+1,1 <β k+1, < <β k+1,m < 0. In particular, if ρ l+1,i = χ r,s or χ 1 r,s, we must have r k. Then, by condition of Lemma 4.1, we have ρ l+1,i 1 is irreducible. However, we know, from [Gol], that ρ l+1,1 ρ l+1,kl+1 1 is irreducible if and only if ρ l+1,i 1 is irreducible for all i =1,...,k l+1. This finishes the proposition. Combining Theorem.6, Propositions 3.3 and 4., we see that ( ) is enough to give irreducibility for G = Sp n (F). We note the similarity of this to 3 1 in Theorem 3.1. [Jan1] (though both the hypotheses and techniques used there are different). We now give another situation where ( ) implies ( ). We return to the more general G of Section 1. Remark 4.3. Suppose that λ = λ 0 e ν λ,h Pmin ( ) with λ 0 = 1. Then, i MΦ1 A(λ 1 ) is irreducible. Proof. As noted earlier, by the construction of Φ 1, we have (ν λ1 ) MΦ1 = 0 and (ν λ1 ) MΦ 1 = νλ1. Thus, i MΦ1 A(λ 1 )=(i MΦ1 A(1)) e ν λ 1,H PΦ1 ( ). Since the trivial representation induces irreducibly, we see that i MΦ1 A(λ 1 )is irreducible. Remark 4.4. The same proof shows ( ) implies ( ) for the classical group SO n+1 (F ). However, as the following example shows, we do not have ( ) implies ( ) in general. Let G be the classical group SO 4 (F) with Fp-adic; ψ a nontrivial character of F with ψ =1 F (1 F the trivial character of F ). Set π =

DEGENERATE PRINCIPAL SERIES 81 i GA (ψ 1 F ). If α 1 = e 1 e, α = e 1 + e denote the two simple roots, then both M 1 = GL (F ) and M = GL (F ). For M 1,( ) requires ψ 1 F and 1 1 F ψ 1 to be irreducible. For M,( ) requires ψ 1 1 F and 1 F ψ 1 to be irreducible. In short, ( ) holds. Consider ( ). For ( ), we may take λ 1 = ψ 1 and have M Φ1 = G. Thus, ( ) holds if π is irreducible. However, an R-group calculation (or a Hecke algebra calculation if ψ is unramified) shows π is reducible. Thus, in general, ( ) need not imply ( ). 5. The example of degenerate principal series for Sp n (F). In this section, we apply the results of the preceding sections to the example of degenerate principal series for Sp n (F). These examples show that, in general, the converse to Theorem.6 does not hold. We start by recalling some well-known reducibility results for GL (F ) and Sp 1 (F)=SL (F). Lemma 5.1. Induced representations for GL (F ), SL (F) have the following reducibility points: (1) For GL (R), χ 1 χ has the following reducibility points: (a) χ 1 = k+1 χ, k Z (b) χ 1 =( k sgn)χ, k Z \{0}, where sgn denotes the unique order two character of R. It is irreducible everywhere else. () For SL (R), χ 1 has the following reducibility points: (a) χ = k+1, k Z (b) χ 1 = k sgn, k Z. It is irreducible everywhere else. (3) For GL (F ) with Fp-adic, χ 1 χ is reducible if and only if χ 1 = χ or χ 1 = 1 χ. (4) For SL (F) with Fp-adic, χ 1 is reducible if and only if χ = 1,, or χ is a nontrivial character with χ =1. Corollary 5.. Let M = GL k (F ) Sp l (F) be a standard Levi subgroup for G = Sp k+l (F). Set π = i GM ((χ det k ) triv l ) for χ a character of F. Assume k + l>1. Write χ = α χ 0, with α R and χ 0 unitary. Then, Theorem.6 gives us the following irreducibility points: (1) χ 0 1,F =Ror p-adic π is irreducible for all α R. () χ 0 =1,χ 0 1,Fp-adic

8 CHRIS JANTZEN π is irreducible if { k +1 α, k+,..., k 1 }. If k =,α =0is also an irreducibility point. (3) χ 0 =1,Fp-adic π is irreducible if { k +1 α, k+,..., k 1 } { l+ k+1 1, l+ k+1,...,l+ k 1 } +1. If k =,α =0is also an irreducibility point. (4) χ 0 =1,χ 0 1,F =R πis irreducible if Z if k =1 1 α +Z if k = Z if k>. (5) χ 0 =1,F =R πis irreducible if 1 Z if k =1 1 α +Z if k = Z if k>. 1 Proof. By Theorem.6 and Propositions 3.3 and 4., we have irreducibility if ( ) holds. Now, for χ = α χ 0,wehave k+1 k+1 α+ λ=( α+ χ 0 +1 k 1 α+ χ 0 χ0 ) ( l l+1 1 ). Therefore, by Lemma 4.1, ( ) holds if the following representations are all irreducible: 1. α+ k+1 +i χ 0 l+j, 0 i k 1, 0 j l 1,. α+ k+1 +i 1, 0 i k 1, 3. α+ k+1 +i χ 0 α+k 1 j χ 1 0, 0 i<j k 1, 4. l+j α+k 1 i χ 1 0, 0 i k 1, 0 j l 1.

DEGENERATE PRINCIPAL SERIES 83 It is now a straightforward matter to use the known reducibility points for GL (F ) and Sp 1 (F)=SL (F), listed in Lemma 5.1, to obtain the corollary. At this point, it is natural to ask whether the irreducibility points given in Corollary 5. are actually all of the irreducibility points. The answer is no. This may be seen by comparing the results in Corollary 5. with known results for degenerate principal series (for Fp-adic, see Theorem 4.3, [Jan]; for F = R, k = 1, see Proposition 10.3 [Cop] or Theorem.6 [Tho]; for F = R, k = n, see Theorem 4.1 [Lee] and Theorem 1 [K-R]). 6. Further examples. In this section, we give two additional examples. The first is an application of the irreducibility criteria to degenerate principal series for G (F ). In the second example, we show how some of the ideas used in this paper can be applied to show irreducibility for a degenerate principal series representation of Sp n (F) not covered by Corollary 5.. For G = G (F ), charf = 0, let α 1 denote the long simple root and α the short simple root. Let s i denote the simple reflection associated to α i and P i = P min,s i =M i U i the corresponding maximal parabolic subgroup. Note that both M 1 = GL (F ) and M = GL (F )([Sha]). We choose the realization of G which has ˇα 1 (t) = diag(t, t 1 ) and ˇα = diag(t 1,t ). (More on the structure of G may be found in [Kim] or[m].) We note that with this realization, if χ 1 χ is a character of A, we have s 1 (χ 1 χ )=χ χ 1 and s (χ 1 χ )=χ 1 χ 1 χ 1. Further, we have i M1A(χ 1 χ ) = χ 1 χ and i MA(χ 1 χ ) = χ χ 1 χ 1. We shall apply the irreducibility criteria of Theorem.6 to the degenerate principal series representations i GM1 (χ det ) and i GM (χ det ). We note that for i GM1 (χ det ), we have λ = 1 1 χ χ; for i GM (χ det ), λ = χ 1 χ (noting that ima(χ 1 χ) = 1 1 χ χ, which has χ det as its Langlands subrepresentation). First, we claim that ( ) holds for i GM1 (χ det ) and i GM (χ det ). Suppose λ 1 = w 1 λ = s1 χ 1 s χ, with s 1,s Rand χ 1,χ unitary. If Φ 1 = {α 1,α } (so that M Φ1 = G (F )), we would have to have s 1 = s =0. AlookatW M1A ( 1 χ 1 χ) and W M A (χ 1 χ) tells us this is not the case for degenerate principal series representations. Thus, M Φ1 = A, M 1, or M.IfM Φ1 =A, there is nothing to prove. If not, M Φ1 = GL (F ). Then, ( ) follows from the fact that s χ 1 s χ is always an irreducible representation of GL (F ). Since ( ) implies ( ), it remains to check when ( ) holds. First, we note that

84 CHRIS JANTZEN W M1M1 A = {s,s s 1 s }, W M1M A = {1,s 1 s,s 1 s s 1 s }, W MM1 A ={1,s s 1,s s 1 s s 1 }, W MM A = {s 1,s 1 s s 1 } (a straightforward calculation using the table in Section [Kim]). Thus, for i GM1 (χ det ), ( ) holds if the following representations are all irreducible: i M1A( 1 χ 1 ) = 1 χ 1 (s W M1M1 A ) i M1A( 1 1 χ 1 ) = 1 1 χ 1 (s s 1 s W M1M1 A ) i MA( 1 χ 1 χ) = 1 χ 1 (1 W M1M A ) i MA( 1 1 χ) = 1 χ 1 χ 1 i MA( 1 χ 1 1 ) = 1 1 χ 1 (s 1 s W M1M A ) (s 1 s s 1 s W M1M A ). Similarly, for i GM (χ det ), ( ) holds if i M1A(χ 1 χ) =χ 1 χ (1 W M M 1 A ) i M1A( 1 χ 1 χ 1 ) = 1 χ 1 χ 1 (s s 1 W MM1 A ) i M1A( 1 χ 1 χ ) = 1 χ 1 χ i MA( 1 χ χ ) =χ 1 χ 1 i MA( 1 χ 1 1 χ) = 1 χ χ (s s 1 s s 1 W MM1 A ) (s 1 W MM A ) (s 1 s s 1 W MM A ). It is now a straightforward matter to apply Lemma 5.1 to see when ( ) holds (hence irreducibility occurs) for i GM1 (χ det ) and i GM (χ det ). If F is p-adic and χ = s χ 0 with s R and χ 0 unitary, we get the following: 1. ( ) holds for i GM1 (χ det ) unless χ 0 = 1 and s {± 1,±3,±5}or χ 0 nontrivial with χ 0 = 1 and s = ± 1.. ( ) holds for i GM (χ det ) unless χ 0 = 1 and s {± 1 6,±1,±3},χ 0 nontrivial with χ 0 = 1 and s = ± 1,orχ 0 nontrivial with χ 3 0 = 1 and s {± 1 6,±1}. We note that the reducibility points are known (Theorem 3.1 [M]): For i GM1 (χ det ), all the points listed except χ 0 =1,s=± 3 are reducibility points; for i GM (χ det ), all but χ 3 0 =1,s=± 1 are reducibility points. If 6 F = R and χ = s χ 0 with s R and χ 0 unitary, we get the following: 1. ( ) holds for i GM1 (χ det ) unless χ 0 = 1 and s 1 + Z.

DEGENERATE PRINCIPAL SERIES 85. ( ) holds for i GM (χ det ) unless χ 0 = 1 and s { 1 +Z} {{± 1 6 + Z}\{± 1 6 if χ 0 1}}. The second example is in response to a question from Dihua Jiang. The degenerate principal series representation of Sp 3k+1 (F ) obtained by inducing the trivial representation from the standard parabolic subgroup with Levi factor M = GL k+1 (F ) Sp k (F) is not covered by Corollary 5.. We show that it is irreducible. The key facts needed for our argument are (1) det k 1 k is an irreducible representation of GL k+1 (F ) (for F p- adic see [Zel]; for F = R, see [H]), and () k triv k is an irreducible representation of Sp k+1 (F ) (for Fp-adic, see [Tad]; for F = R, see [Cop] or [Tho]). We note that this argument does not require charf =0 itis enough to assume that if F is p-adic, charf. We show the irreducibility of 1 GL(k+1,F ) triv k by induction on k. More precisely, we show that 1 GL(k+1,F ) triv k = L( k k k, k+1 k+1 k+1,..., 1 1 1 ;1 F 1). The case k = 0 is trivial. In general, we have 1 GL(k+1,F ) triv k ( det k 1 k ) triv k = det k 1 ( k triv k ) = det k 1 ( k triv k ) from () above det k 1 k k triv k 1 = k det k 1 k triv k 1 from (1) above = k k det k 1 trivk 1 by (1) above k k ( k 1 GL(k 1,F ) ) triv k 1 = ( k k k ) (1 GL(k 1,F ) triv k 1 ) = ( k k k ) L( k+1 k+1 k+1,..., 1 1 1 ;1 F 1) by inductive hypothesis ( k k k ) ( k+1 k+1 k+1 ) ( 1 1 1 ) (1 F 1), which has unique irreducible subrepresentation L( k k k,..., 1 1 1 ;1 F 1) by the Langlands classification. Therefore, 1 GL(k+1,F ) triv k has L( k k k,..., 1 1 1 ;1 F 1)

86 CHRIS JANTZEN as unique irreducible subrepresentation. Since 1 GL(k+1,F ) triv k is unitary, it decomposes as a direct sum of irreducible representations. Therefore, we must have 1 GL(k+1,F ) triv k = L( k k k,..., 1 1 1 ;1 F 1), as claimed. We remark that in the case Fp-adic, the proof of this fact was originally done using Jacquet module arguments. (The proof is not given explicitly in [Jan] as it is essentially the same as the proof of Lemma 4. there.) The argument above has the advantages that (a) it works for F = R as well, and (b) it is significantly shorter. References [B-Z] I. Bernstein and A. Zelevinsky, Induced representations of reductive p-adic groups I, Ann. Sci. École Norm. Sup., 10 (1977), 441-47. [B-W] A. Borel, and N. Wallach, Continuous cohomology, discrete subgroups, and representations of reductive groups, Princeton University Press, Princeton, 1980. [Cop] M. Copper, The Fourier transform and intertwining operators for certain degenerate principal series representations of Sp(n; F), F= R or C, preprint. [Gol] D. Goldberg, Reducibility of induced representations for Sp(n) and SO(n), Amer. J. Math., 116 (1994), 1101-1151. [H-C] Harish-Chandra, Harmonic analysis on reductive p-adic groups, Proc. Sympos. Pure Math., 6 (1973), 167-19. [H] R. Howe, Some simple examples in representation theory in semisimple Lie groups, Jber. d. Dt. Math.-Verein., 96 (1995), 131-160. [Jan1] C. Jantzen, Degenerate principal series for symplectic groups, Mem. Amer. Math. Soc., 488 (1993). [Jan], Degenerate principal series for symplectic and odd-orthogonal groups, Mem. Amer. Math. Soc., 590 (1996). [Kim] H. Kim, The residual spectrum of G, Can. J. Math., 48 (1996), 145-17. [K-R] S. Kudla and S. Rallis, Degenerate principal series and invariant distributions, Isr. J. Math., 69 (1990), 5-45. [Lan] R. Langlands, On the classification of irreducible representations of real algebraic groups, in Representation Theory and Harmonic Analysis on Semisimple Lie Groups (eds. Paul Sally, Jr. and David Vogan), Mathematical Surveys and Monographs, AMS, 31, (1989), 101-170. [Lee] S. Lee, Degenerate principal series representations of Sp(n, R), Comp. Math., 103 (1996), 13-151. [M] G. Muić, The unitary dual of p-adic G, preprint. [Sha] F. Shahidi, Third symmetric power L-functions for GL(), Comp. Math., 70 (1989), 45-73.

DEGENERATE PRINCIPAL SERIES 87 [Sil] A. Silberger, The Langlands quotient theorem for p-adic groups, Math. Ann., 36 (1978), 95-104. [Tad1] M. Tadić, Representations of p-adic symplectic groups, Comp. Math., 90 (1994), 13-181. [Tad], On reducibility of parabolic induction, Isr. J. Math., to appear. [Tho] H. Thorleifsson, The composition series of Sp(n, R), n, induced from onedimensional representations of a certain maximal parabolic subgroup, preprinted in Math. Göttingesis, 10 (1993). [Zel] A. Zelevinsky, Induced representations of reductive p-adic groups II, On irreducible representations of GL(n), Ann. Sci. École Norm. Sup., 13 (1980), 165-10. Received March 1, 1997 and revised September 10, 1997. 500 Lincoln Ave Fox River Grove, IL 6001 E-mail address: jantzen@math.uchicago.edu