98m:11125 11R39 11F27 11F67 11F70 22E50 22E55 Gelbart, Stephen (IL-WEIZ); Rogawski, Jonathan (1-UCLA); Soudry, David (IL-TLAV) Endoscopy, theta-liftings, and period integrals for the unitary group in three variables. Ann. of Math. (2) 145 (1997), no. 3, 419 476. FEATURED REVIEW. This article describes the structure of local and global L-packets for the quasisplit unitary group U(3) from the points of view of endoscopy, theta lifting and period integrals. It provides the clearest evidence to date of the connections one should expect to find for general reductive groups among these three notions which lie at the heart of the theory of automorphic representations. It is also of interest because of its connections to the theory of algebraic cycles on Picard modular surfaces. The research represents the culmination of a series of papers by the authors. Understanding this paper requires substantial preparation. Some of this is provided by Gelbart s survey article [in Theta functions: from the classical to the modern, 129 174, Amer. Math. Soc., Providence, RI, 1993; MR 94f:22023]. On the specific topics of endoscopy, theta liftings and period integrals, we also mention the following introductory articles by J. G. Arthur [in Representation theory and automorphic forms (Edinburgh, 1996), 433 442, Proc. Sympos. Pure Math., 61, Amer. Math. Soc., Providence, RI, 1997; MR 98j:22029], D. Prasad [in Theta functions: from the classical to the modern, 105 127, Amer. Math. Soc., Providence, RI, 1993; MR 94e:11043], and H. M. Jacquet [in Representation theory and automorphic forms (Edinburgh, 1996), 443 455, Proc. Sympos. Pure Math., 61, Amer. Math. Soc., Providence, RI, 1997; MR 98m:22024]. To state the results of the present paper, it is necessary to recall some basic concepts. According to Langlands theory, one expects to attach a stable trace formula to a general connected reductive group defined over a global field. The spectral side of the trace formula involves a sum over L-packets of representations of the given group, as well as its elliptic endoscopic subgroups. In the case of interest, G = U(3), the L-packets are understood, as is the stable trace formula, thanks to Rogawski s work [Automorphic representations of unitary groups in three variables, Ann. of Math. Stud., 123, Princeton Univ. Press, Princeton, NJ, 1990; MR 91k:22037]. In this case, there is a unique elliptic endoscopic group, namely, H = U(2) U(1), where U(2) denotes the quasisplit unitary group in two variables. All of these unitary groups are defined with respect to some fixed quadratic
extension E/F of global fields. For each idele class character γ of E which extends the class field theory character of E/F, there is an associated embedding of L-groups L H L G which is used to map each L-packet ρ of H to an L-packet Π(ρ) of G. The L-packets of primary interest are referred to as (cuspidal) endoscopic L-packets. These are the cuspidal L-packets Π(ρ), for which ρ = ρ 2 ρ 1 is a cuspidal L-packet (consisting of infinite-dimensional representations). It is shown that each endoscopic L-packet contains a unique generic representation. The analogous local result is also proven. If Π is an endoscopic L-packet, then its fiber Π under endoscopic transfer is a set of at most three L-packets for H. In fact, Rogawski has shown that there is a natural way to identify the set S Π = Π {1} with one of the groups Z/2 or Z/2 Z/2 so that the cuspidal multiplicity of π Π is given by a formula m(π) = S Π 1 ρ S Π ρ, π, for some character ρ ρ, π of S Π. This is compatible with the multiplicity formulas outlined by Arthur for general groups. The multiplicity-one property of U(3) implies that m(π) = 0 or 1. The endoscopic lifting of ρ = ρ 2 ρ 1 to Π(ρ) is characterized by an L-function identity which relates the L-function of a twist of π Π(ρ) to the product of L-functions of corresponding twists of the standard L-functions associated to the base change lifts, with respect to E/F, of ρ 2 and ρ 1 to representations of GL(2) and GL(1), respectively. In addition to endoscopic lifting from H to G, one may construct representations of G via theta lifting from unitary ( groups ) in two variables. If d F, we define a matrix Φ d = d 0 and let H 0 1 d denote the associated unitary group. Then the isomorphism class of H d is determined by the class of d modulo norms from E. The choice of a nontrivial character ψ of A/F together with the choice of γ determines a theta correspondence between cuspidal representations σ of H d and certain representations Θ ψ (σ, γ) of G. When σ does not correspond to a representation of G, then the theta lift is considered to be zero. The group G lies in a tower of unitary groups U(2n + 1) and one may consider theta lifting from H d to any of the groups in the tower. The philosophy of towers suggests that, given σ, the first occurrence of a nonzero theta lift of σ in the tower must be cuspidal. Indeed, the authors showed in a previous paper that if the theta lift of σ to U(1) is zero then the theta lift to G must be cuspidal, though possibly zero. In the present paper, they show that if the theta lift of σ = v σ v to U(1) is zero then the theta lift Θ ψ (σ, γ) is nonzero precisely when the local Howe lift of each component σ v is nonzero.
Much of the above discussion has an obvious adaptation to the local case, which we will not provide here. In short, if E/F is local, we are interested in L-packets of irreducible, admissible representations of the quasisplit group G(F ) = U(3, E/F ) which are obtained from the elliptic, endoscopic group H by functorial transfer. Such L-packets will be called endoscopic L-packets. There are local multiplicity pairings ρ, π which are related to the global pairing by a product formula. Given an irreducible, admissible representation σ of the local group H d (F ), we will let H ψ (σ, γ) denote the Howe lift of σ to G(F ). We now describe the connection between endoscopic lifting and Howe lifting. Suppose that π = H ψ (σ, γ) belongs to a local endoscopic L-packet Π. Then it is shown that there exists ρ Π such that ρ 1 = γ E 1, where E 1 = U(1). On the other hand, if π is any representation which belongs to an endoscopic L-packet Π and if ρ Π satisfies ρ 1 = γ E 1 then there exist d, σ, ψ such that π = H ψ (σ, γ). Moreover, d is unique modulo norms. This leads to the definition ε ρ (π) = 1 if d is a norm, and ε ρ (π) = 1 if d is not a norm. It is proven that, in fact, ε ρ (π) = ρ, π. A more general statement of these results which applies to twists of Howe lifts by characters of E 1 is actually proven. The p-adic endoscopic L-packets may be parametrized in terms of Howe lifts as follows. Recall that both the functorial lifting and the theta lifting depend on the choice of the character γ. For simplicity, we are using the same choice of γ for both purposes. (In the notation of the paper, we are assuming µ = γ 1.) Fix a nonnorm d 0. Then there is an analogue of the Jacquet-Langlands correspondence between the quasisplit group H 1 and the non-quasisplit group H d0. Assume ρ 2 is an L-packet on H 1. Let ρ 2 be the corresponding L-packet on H d0, if it exists, and let ρ 2 = otherwise. The authors show that the endoscopic L-packet associated to ρ = ρ 2 ρ 1, with ρ 1 = γ E 1, is identical to the collection of all nonzero Howe lifts H ψ (σ, γ) such that σ ρ 2 ρ 2. Moreover, the set of all such σ (with nonzero Howe lift) is precisely described. Once again, the actual theorem proven applies more generally to twists of Howe lifts. Returning to the global case, assume that π = v π v belongs to an endoscopic L-packet Π(ρ) and γ E 1 = ρ 1. Let ε ρ (π) = v ε ρ v (π v ) and assume that ε ρ (π) = 1. Then we may choose d F which is a norm at precisely those places v at which ε ρv (π v ) = 1. We may then form an admissible representation σ = v σ v of H d (A) by choosing σ v such that π v = H ψv (σ v, γ). It is shown that m(π) = 1 if and only if m(σ) = 1, where m(σ) is the cuspidal multiplicity of σ with respect to the spectrum of H d. On the other hand, if ε ρ (π) = 1 then m(π) = 0. We now turn to the relation between the theta lifts and endoscopic
lifts considered above and the nonvanishing of certain period integrals. The period integrals of interest arise as period integrals on Picard modular surfaces over algebraic cycles corresponding to unitary groups in two variables. They are defined by P (ϕ, η, d) = ϕ(h)η 1 (det h)dh, H d (F )\H d (A) where ϕ is an automorphic form in the space of the cuspidal automorphic representation π of G, the character η is defined on E 1 \E 1 A, and we have fixed an embedding of H d in G. These integrals are analogous to the period integrals used by Harder, Langlands and Rapoport to prove Tate s conjectures for Hilbert modular surfaces. If P (ϕ, η, d) is nonzero for some d and ϕ, we say π is η-distinguished. An endoscopic L-packet which contains a (cuspidal) η-distinguished element is also said to be η-distinguished. It is proven that if Π is such an L-packet, then Π must contain an element ρ such that η = ρ 1. If π Π is cuspidal, then the global condition of π being ρ 1 -distinguished is shown to be equivalent to the local conditions ρ v, π v = 1, for all v. In this case, the L-function L(s, π η 1 ) must have a pole at s = 1. Given an η-distinguished representation π, we may choose ϕ = v ϕ v such that P (ϕ, η, d) 0 and then vary one of the components ϕ v. This yields a local period functional on the space of π v, that is, a nonzero linear functional λ on the space of π v such that λ(π v (h)v) = η v (det(h))λ(v), for all h H d (F v ). An irreducible, admissible, infinitedimensional representation of G(F v ) is said to be η v -distinguished if it admits a period functional, for some H d. It is shown in the p-adic case that such a representation must lie in an endoscopic L-packet Π(ρ v ) such that η v = ρ v1 and ρ v, π v = 1, and, conversely, that these conditions characterize the η v -distinguished representations. It is also shown that a cuspidal representation π is η-distinguished if and only if it is η v -distinguished for all v. On the other hand, if one fixes d and only considers periods relative to this choice, then it is possible to find π such that P (ϕ, η, d) 0 even though there exist (nonzero) period functionals relative to H d at each place of F. Putting together some of the results mentioned above, one deduces that the cuspidal representations π with nonzero periods are those cuspidal representations which are theta lifts of cuspidal representations σ of the quasisplit group H 1. Suppose, in this case, that σ E is the base change lift of σ to GL(2) /E. The last result of the paper is
the formula P (ϕ, γ E 1 A, d) = L S (1, σ E γ 3 ) v S C v (ϕ v ), in which S is a finite set of places of F, the C v s are period functionals, and the components of the test function ϕ = v ϕ v outside of S are held fixed. Besides the work of Harder, Langlands and Rapoport on Hilbert modular surfaces, one of the most obvious precursors of this paper is Waldspurger s work on the Shimura correspondence. Many of the connections among theta lifting, special values of L-functions and period integrals were already evident in the work of Waldspurger, as well as in many subsequent applications of Jacquet s theory of relative trace formulas on GL(2). The present paper fully integrates the theory of endoscopy into this mix and it greatly increases our understanding of the theory of automorphic representations for U(3). Jeff Hakim (1-AMER)