53'(M) (resp. 8'(M)) consists of all distributions on M (resp. distributions on M

Transcription

1 APPLICATIONS OF THE RADON TRANSFORM TO REPRESENTATIONS OF SEMISIMPLE LIE GROUPS* BY SIGURDUR HELGASON MASSACHUSETTS INSTITUTE OF TECHNOLOGY Communicated by Lars V. Ahlfors, April 21, 1969 Abstract.-From the point of view of the duality between points and horocycles in a symmetric space, the counterparts to the spherical functions on the symmetric space are the conical distributions on the manifold of horocycles. While the conical functions are closely related to certain finite-dimensional representations of semisimple Lie groups, in the present work the conical distributions are found to play various roles in the principal series of infinitedimensional representations of these groups. 1. Introduction.-The manifold of horocycles in a symmetric space carries a certain class of distributions introduced in an earlier paper' under the name conical distributions. In section 3 we indicate how analytic continuation techniques can be used to determine these distributions. The result is applied to give a complete solution (Theorem 4.6) of the problem of constructing the intertwining operators to irreducible equivalent representations from the spherical principal series of a semisimple Lie group2 G. The conical distributions obtained also provide an analytic approach to the irreducibility question for the spherical principal series whereby irreducibility is expressed directly by means of Harish-Chandra's c-function. For complex G a criterion was obtained by Parthasarathy, Rao, and Varadarajan;3 by an extension of their purely algebraic method, Kostant4 obtained an irreducibility criterion for all real G, although in a form quite different from that of Corollary 4.3. Finally, when the spherical principal series is realized on appropriate distribution spaces (over the space of horocycles), the conical distributions provide the highest weight vectors (cf. Corollary 4.4). The results show that both the numerator and the denominator of the c-function have their individual significance (cf. Theorem 3.2 and Corollary 4.3). 2. Notation.-If M is a manifold, following the book of Schwartz,5 we denote by 8(M) the set of C- functions on M and by D(M) the set of C- functions on M of compact support, both spaces taken with their usual topologies. The dual 53'(M) (resp. 8'(M)) consists of all distributions on M (resp. distributions on M of compact support). Both spaces will be taken with the strong topology. Let G be a connected semisimple Lie group with finite center, G = KAN an Iwasawa decomposition of G where K is compact, A abelian, and N nilpotent. Let M be the centralizer of A in K, 0 the Cartan involution of G which is identity on K, put JN = ON and let g, f, a, n, m, and n- denote the respective Lie algebras of the groups above. Let log denote the inverse of the map exp: a -0 A. Let M' denote the normalizer of A in K, W the Weyl group M'/M, and B the coset space K/M. Let a* denote the dual space of a, 2 c a* the set of restricted 643

2 644 MATHEMATICS: S. HELGA SON PROC. N. A. S. roots; if a e 1, let g.1 denote the corresponding root space and put ma = dim 9a. Let 2;+ denote the set of positive restricted roots such that n= wev + Let Io+ = {a e + '/2a f 2} and let 2- and Io- denote the set of negatives of the elements of 2+ and 2o+, respectively. Let a,* denote the complex dual of a and (, ) the bilinear form on g and a,* induced by the Killing form of g. 3. Conical Distributions.-Consider now the symmetric space X = G/K and the space6 : = G/MN of all horocycles in X. A distribution on Z is called conical if (1) it is MN-invariant and (2) it is an eigendistribution of each G-invariant differential operator on S. Since K\G/K = AIW, MN\G/MN = A X W, one can, because of the analogy with spherical functions, expect a natural parametrization of the set of conical distributions in terms of a,* X W. Theorem 3.3 is a partial verification of this. For each s e W, fix m, e M' in the coset s and put Is = mmn. If :. denotes the orbit MNA. 4, the Bruhat lemma implies Z = U ri. (disjoint union). If N.' = N n mnvm'-, the mapping (n,', a) n/a-t, is a diffeomorphism of N8' X A onto 4s, giving 2, a natural measure dv. Given t e 4 there exists a - unique a(t) e A such that t e MNa(t) i,; for X e a,* we consider the functional V X Ax0(0~e(isX+sp)(log a(6)) dv(t) ( where p = 2 2a >0 maa. LEMMA 3.1. The integral (1) converges absolutely for all sp e 53(A) if and only if Re((a, ia)) > 0 for a e 2+ n S (2) Re denoting real part. If (2) is satisfied, ",. is a conical distribution on E. The function X -tv, is holomorphic in the tube (2) with values ind One of our main results is that it can be extended to a meromorphic function on the entire a,* and its values and "residues" are still conical distributions. These residues are certain transversal derivatives of other 'I"t,8' (corresponding to other MNA-orbits in the closure of Es). Consider the function cs(x) = d.(x)e,,(x), where d,(x) = II 2 ar)p((ix, ao)), aez, e,,(x)~= a I 02(Z2 r (ix, ao))j r + m2a + (ix,ao))]. Here ao = (a, a)-la and r is the usual Gamma function. For the Weyl group element s = s* which interchanges 2 + and 2-, c,(x) is up to a constant factor equal to Harish-Chandra's c-function.7 THEOREM 3.2. For each s e W, the mapping X -- d,(x) = *,s

3 VOL. 63, 1969 MATHEMATICS: S. HELGASON 645 extends from (2) to a holomorphic function on the entire space a,*. Its values are conical distributions on. Let D(X) and D(X) denote the algebras of G-invariant differential operators on X and X, respectively, and D(A) the set of left invariant differential operators on A. Let p e a,* and x,: D(r) -* C the homomorphism obtained by extending to a homomorphism of D(A) into C and combining with the isomorphism8 of ;& D(A) onto D(z). Let )x'(e) = {I e '(M) IDT = xix-p(d)i for De D( }, and let X), denote the Hilbert space of locally square integrable functions in VO), with norm 4 [fkj4 (kmn) I 2dkJI/2 dk being Haar measure on K. Each conical distribution lies in one of the spaces THEOREM 3.3. not zero at X. Suppose sx $ X for all s $ e in W and assume the function es* is Then the linear combinations E a8%*,8 a, e C aew constitute all the conical distributions in DN'. The following lemma plays a crucial role in the proof of Theorem 3.2. If g e G, let H(g) e a, k(g) e K be determined by g = k(g) exp H(g)n (n e N), and write m* for m,*. If g e NMAN, let #A(g) e N be determined by g = ft(g)man (m e M, a e A, n en). LEMMA 3.4. Suppose G/K has rank one and put IZ 12 = -(Z, OZ) for Z Q Then if = exp X exp Y (X e g Y e.-2a),we have ep(h(a)) = [(1 + C IX 12)2 + 4c Y 12]1/4("a + 2m2a) ep(h(f&(m))-(a)) = [c2 X c Y j2]1/4(ma+2m2a). The constant c is given by c-' = 4(ma + 4m2a,.). Remark: The first formula provides a simplified way of computing Harish- Chandra's c-function c(x) = fij-e (i+p)(h(i))d in the rank-one case.9 because It also gives an explicit formula for the Poisson kernel P(gK, km) = e-2plh(g-1k)) P(aK, k(ft)m) = e2p(loga) e2p(h(a)fh(a-l1a)), a e A, e N 4. Intertwining Operators, Highest Weight Vectors, and Irreducibility Criteria. Let TXi and x>, denote the natural representations of G on 5Cx and 0',, respectively. Then TX is equivalent to the representation of G induced10 by the character man --i(lo a) of MAN; moreover, rv, is irreducible if and only if al, is irreducible (no proper closed invariant subspace 0 0). We refer to the set of

4 646 MA THEMA TICS: S. HELGA SON PROC. N. A. S. representations Tx as the spherical principal series. Let 1'o denote the "unique" vector in C;, fixed under K. Let f -- f denote the Radon transform on X and 'I -+ 'its dual given by +(f) = +(f) (f e D(X), e VD'( )). We define A irreducible if TX is irreducible; X cyclic if,6o is a cyclic vector for r,; X simple if I - is injective on i,. LEMMA 4.1. Let X e a,* be arbitrary. Then (i) X is cyclic (=) -X is simple; (ii) X is irreducible (=) X and - X are simple. LEMMA 4.2. For each X e a,* and each s e W, (1X,J) is a function on X and (Ix,,)'(naK) = ce (X)e(iSx+ p)(log a) n e N, a e A, (3) where c is a constant. On the basis of this relation, we find that X is not simple if es*(x) = 0. It also implies that if e8*(x) # 0 and X not simple, then for each s W. sa is not simple. Now Kostant has proved4 that X is cyclic if Re((iX, a)) < 0 for a E E+ (4) Hence we deduce from Lemma 4.1 that X is simple if and only if e8*(x) 5 0, (5) and we deduce COROLLARY 4.3. X is irreducible if and only if e8*(x)e8*(-x) $ 0. Remark: The "if" part required (4), which follows from Lemma 4.1 (i) except for the justification of certain interchange of limit and integration. This justification has only been carried out for the strict inequality in (4) and also for rank (X) = 1, so the reasoning indicated above does not constitute an independent proof of the irreducibility criterion. A vector ' e J%)' is called a highest weight vector if it is an eigenvector of ix(a) and fixed under!x(n). From Theorem 3.3 and Lemma 4.2, we deduce COROLLARY 4.4. Let X be as in Theorem 3.3. Then the scalar multiples of the vectors Id,, constitute precisely the highest weight vectors in D>x'. With the natural pairing (, ) between JCX and D -' whereby the representation Tx becomes contragredient to Tx, we have )',e(4)) = (d) X I',81 Oa,81),4 f (Z), 5.E W, (6) where X denotes convolution11 on S. These formulas are infinite-dimensional analogs of the formula12 relating conical functions and finite-dimensional conical representations. We now transfer the representation space of Ax from 5x' to V'(B). Let NST Xfn (mj-'nm,) and let da, and da denote Haar measures on N, and A, respectively.

5 VOL. 63, 1969 MA THEMA TICS: S. HELGASON 6;47 LEMMA 4.5. The mapping S - I given by it(0) =,fi1mja(kamn)e(+p) (loga) da)ds(km), )e 2D(E) is a bijection of D'(B) onto Dx' mapping L2(B) onto ICE. Under this map 'x,8 corresponds to the distribution SX,s: F -- d,(x)-' fit8f(msk(fts)m)e -i+p)(h(f8)) dis, F e 8(B), defined for all X e a,* by analytic continuation. The distributions S,8 now give the construction of the interwining operators. Let Si X S2 denote the convolution product of distributions on B induced by convolution on K. Although in general this product only commutes with the action of K, we see now that for S2 = S,,,, we get commutation with the action of G. THEOREM 4.6. Assume X irreducible. The convolution operator S -* S X S8A,8-' on D'(B) induces a bijection Ax,.: D ' -o.' which intertwines the representations f,, and fx. If X is real, then Ax,, gives a unitary operator 5C, onto 5C8x. Note added in proof: Since the completion of this paper, G. Schiffman has kindly sent me a copy of his thesis. It contains a statement of Lemma 3.4, the last part of Theorem 4.6, and several other results on intertwining operators. * This work was supported in part by the National Science Foundation (NSF GP 9003). 1 Helgason, S., "Radon-Fourier transforms on symmetric spaces and related group representations," Bull. Am. Math. Soc., 71, (1965). 2 For complex G the problem is settled by R. A. Kunze and E. M. Stein ("Uniformly bounded representations, III," Am. J. Math. 89, (1967)), and G. Schiffmann ("Intdgrales d'entrelacement," Compt. Rend., 266, (1968)). Schiffmann also treats the case of G which split over R and for general G reduces the problem to the rank-one case which, however, is left open; see also Gelfand, I. M., M. I. Graev, and I. I. Pyatetskii-Shapiro, Representation Theory and Automorphic Functions (Philadelphia: W. B. Saunders Co., 1969), p Parthasarthy, K. R., R. Ranga Rao, and V. S. Varadarajan, "Representations of complex semisimple Lie groups and Lie algebras," Ann. Math., 85, (1967). 4 Kostant, B., "On the existence and irreducibility of certain series of representations," Bull. Am. Math. Soc., to appear. 5Schwartz, L., Theorie des Distributions (Paris: Hermann, 1966). 6 Helgason, S., "Duality and Radon transform for symmetric spaces," Am. J. Math., 85, (1963). 7 Cf. Harish-Chandra, "Spherical functions on a semisimple Lie group I," Am. J. Math., 80, 303 (1958); Gindikin, S. G., and F. I. Karpelevic, "Plancherel measure for Riemannian symmetric spaces of nonpositive curvature," Soviet Math., 3, (1962). See reference in ref. 6, t4. 9See Harish-Chandra reference in ref. 7. l0 Here, "induced" is meant in the sense of Mackey, G. W., "Induced representations of locally compact groups I," Ann. Math., 55, (1952), and Bruhat, F., "Sur les repr6sentations induites des groupes de Lie," Bull. Soc. Math. France, 84, (1956). 1l Helgason, S., "A duality in integral geometry on symmetric spaces," Proc. U.S.-Japan Seminar in Differential Geometry, Kyoto (1965), p Cf. reference in ref. 1, Theorem 2.6.