SMOOTH TRANSFER OF KLOOSTERMAN INTEGRALS

Transcription

1 SMOOTH TRANSFER OF KLOOSTERMAN INTEGRALS HERVÉ JACQUET Abstract We establish the existence of smooth transfer between absolute Kloosterman integrals and Kloosterman integrals relative to a quadratic extension. Contents. Introduction Orbital integrals in M(m m, F Weil formula: The split case Inversion formula for the orbital integrals Orbital integrals in H(n n, E/F Weil formula: The twisted case Inversion formula in the Hermitian case Smooth matching Application to the fundamental lemma Concluding remarks References Introduction Let E/F be a quadratic extension of local non-archimedean fields, let η : F {±} be the corresponding quadratic character, and let ψ : F C be a nontrivial character. Let N m be the subgroup of upper triangular matrices in GL(m with unit diagonal, and let A m be the group of diagonal matrices. We define a character θ : N m (F C by θ(n = ψ ( n i,i+. The group Nm (F N m (F operates on GL(m, F by g t n gn 2, and the orbits that intersect A m (F form a dense open subset. We define the diagonal orbital integrals of a smooth function of compact support on GL(m, F: (, ψ : a := ( t n an 2 θ(n n 2 dn dn 2. ( N m (F N m (F DUKE MATHEMATICAL JOURNAL Vol. 20, No., c 2003 Received 28 May Revision received 5 November Mathematics Subject Classification. Primary F70; Secondary F72, L05. Author s work partially supported by National Science Foundation grant number DMS

2 22 HERVÉ JACQUET Likewise, we let H(m m, E/F be the space of Hermitian matrices m m and set S m (F = H(m m, E/F GL(m, E. We define a character n θ(nn of N m (E by θ(nn = ψ ( (n i,i+ +n i,i+. The group N m (E operates on S m (F by s t nsn, and the orbits that intersect A m (F form a dense open subset of S m (F. We define the diagonal orbital integrals of a smooth function of compact support on S m (F: (, E/F, ψ : a := ( t nanθ(nn dn. (2 N m (E We say that matches for ψ if for every a A m (F, where (, ψ : a = γ (a, ψ (, E/F, ψ : a, γ (a, ψ := η(a η(a a 2 η(a a 2,..., a m if a = diag(a, a 2,..., a m. In this paper we prove that for every there is a matching, and conversely. Suppose that E/F is unramified, and suppose that the conductor of ψ is O F. Let 0 be the characteristic function of the set of matrices with integral entries in M(m m, F. Similarly, let 0 be the characteristic function of the set of matrices with integral entries in H(m m, E/F. The fundamental lemma asserts that the restriction of 0 to GL(m, F matches the restriction of 0 to S m (F. This has been established by Ngô 0] in the case of positive characteristic. In general, if matches, then there are similar relations between the other orbital integrals (see 3], 4]. Within the context of the relative trace formula, the problem at hand is the analogue of the transfer problem for the ordinary trace formula. Indeed, when E/F is an extension of function fields, the result presented in this paper, together with the fundamental lemma of Ngô 9], 0], implies the existence of global identities of the form ( θ(n n 2 dn dn 2 ( t n ξn 2 N m (F N m (F\N m (F A N m (F A ξ Gl(n,F ( = θ(nn dn N m (E\N m (E A ξ S m (F ( t nξnθ(nn dn. In turn, the identities are a crucial step in extending the results of 5] to GL(m. Our method is first to linearize the problem by replacing GL(m, F and S m (F by the vector spaces M(m m, F and H(m m, E/F, respectively. We then establish a simple relation between the orbital integrals of a function on either space and its Fourier transform. It follows that if matches, then the Fourier transform

3 SMOOTH TRANSFER OF KLOOSTERMAN INTEGRALS 23 of matches the Fourier transform of. This proves the existence of many pairs of matching functions and, in turn, is used to prove the existence of the transfer in a very simple way. Our method is suggested by the work of J. Waldspurger 3]. Waldspurger considers orbital integrals in the context of the Lie algebras and derives the existence of the transfer from the fundamental lemma (assumed to hold and global identities. He uses the Fourier transform on the Lie algebras in an essential way. It is reasonable to ask whether, conversely, the results of the paper can be used to prove the fundamental lemma in the context of the relative trace formula. We answer this question in the affirmative for the case of m = 2, 3 in Section 9. In a forthcoming paper (see 7], we use the previous results to prove the fundamental lemma for all m. 2. Orbital integrals in M(m m, F We now consider the action of N m (F N m (F on M(m m, F given by r t n rn 2. We denote by θ 0 the algebraic character θ 0 : N(F F defined by θ 0 (n = i=m i= n i,i+. We say that an element x M(m m, F (or its orbit is relevant if the character (n, n 2 θ 0 (n n 2 is trivial on the stabilizer of x in N(F N(F. We let i (x, i m, be the determinant of the matrix obtained by removing the last m i rows and the last m i columns of a matrix x. In particular, m (x = det x. The functions i are invariants of the action of N m (F N m (F. For i m, we denote by O i the set defined by i 0. We denote by B m the group of upper triangular matrices in GL(m, by A m the group of diagonal matrices, and by α, α 2,..., α m the simple roots of A m corresponding to B m. We denote by N α the root group corresponding to a root α. We identify the Weyl group of A m with the group W m of permutation matrices, and we let A m be the set of diagonal matrices in M(m m, F. We often drop the index m from the notation. PROPOSITION In M(m m, F the set defined by m = 0, m = 0 contains no relevant orbit. Proof We begin with an elementary lemma. LEMMA Any r M(m m, F can be written (in possibly several ways in the form r = t nwb

4 24 HERVÉ JACQUET with w W m, n N m (F, and b an upper triangular matrix. Proof We let F 0 = {V 0 V V 2 V n } be the canonical flag. Thus V i is the space spanned by the first i vectors of the canonical basis. Let r M(m m, F be given. Consider the sequence of subspaces r V 0 r V r V 2 r V m. It need not be a flag, but, inductively, it is easy to show that there is a flag F = {U 0 U U 2 U m } such that r V i U i, 0 i n. By the standard Bruhat decomposition, F = t nwf 0 for suitable w and n N m (F. Thus ( t nw r V i V i, 0 i n. Since ( t nw r stabilizes F 0, it is an upper triangular matrix. This proves the lemma. To prove our assertion, it suffices then to consider the case of a relevant element of the form r = wb with w W m and b an upper triangular matrix such that det b = 0. Our task is then to show that m (wb 0. We remark that if the column of index i, i < m, of the matrix r is zero, then r is irrelevant because then rn αi = r for all n αi N αi (F. In particular, the first diagonal entry b, of b is nonzero. Then at the cost of multiplying b by a suitable element of N α on the right, we may assume that b,2 = 0. Again, since the second column of b cannot be zero, we must have b 2,2 0. Inductively, we see that we may assume b to have the form b = ( b 0, 0 0

5 SMOOTH TRANSFER OF KLOOSTERMAN INTEGRALS 25 where b is an invertible diagonal matrix of size (m (m. Thus the last row of b is zero. Since wb is relevant, the first m rows of wb cannot be zero. This forces w to have the form w = ( w 0 0, where w is a permutation matrix in GL(m. Now Thus m (wb 0, as claimed. m (wb = det b det w. Now it is elementary that every element x such that det x = 0 but n (x 0 is in the orbit of an element of the form ( x Moreover, x is relevant if and only if x is a relevant element in GL(m, F. Also, a = diag(a, a 2,..., a m A m is relevant if and only if a a 2 a m 0. Now we recall the classification of the relevant orbits in GL(m, F (cf. ], 2]. Every orbit has a unique representative of the form wa with w W and a A m (F. If this element is relevant, then for every pair of positive roots (α, α 2 such that wα 2 = α, for n αi N αi (F we have t n α wan α2 = wa θ 0 (n α n α2 = 0. (3 This condition implies that if α is simple, then α 2 is simple, and conversely. Thus w and its inverse have the property that if they change a simple root to a negative one, then they change it to the opposite of a simple root. Let S be the set of simple roots α such that wα is negative. Then S is also the set of simple roots α such that w α is negative and ws = S. Let M be the standard Levi subgroup determined by S. Thus S is the set of simple roots of M for the torus A, w is the longest element of W M, and w 2 =. This being so, if α 2 is simple, then condition (3 implies α 2 (a =. Thus a is in the center of M. Let w m be the m m permutation matrix with antidiagonal entries equal to. We see that elements of the form g = w m a w m2 a w mr a r,

6 26 HERVÉ JACQUET with m + m m r = m, a, a 2,..., a r F, form a set of representatives for the relevant orbits in GL(m, F. A set of representatives for the relevant orbits in M(n n, F is formed of the same elements, except that a r = 0 is allowed when m r = (and w mr =. We set θ = ψ θ 0. We define the orbital integral of a relevant element x:, ψ : x] = ( t n xn 2 θ(n n 2 dn dn 2 ; the integral is over the quotient of N m (F N m (F by the stabilizer of x. The normalization of the measure is described below. It is convenient to introduce the intermediate orbital integrals. Let m, n be two integers greater than zero. For S (M((m + n (m + n, F, A n GL(n, F, B m M(m m, F, we set n m, ψ : ( ] An 0 := 0 B m ( ( ( ] n 0 An 0 n X Y m 0 B m 0 m ] d X dy. (4 ψ Tr ɛ m n X + Tr Y ɛm n Here ɛ = ɛn m is the matrix with m rows and n columns whose first row is the row matrix of size n, ɛ n = (0, 0,..., 0,, and all other rows are zero. Likewise, ɛ = ɛ n m is the matrix with n rows and m columns whose first column is the column matrix 0 0 ɛ n =..., 0 and all other columns are zero. If X = (x i, j, then the measure d X is the tensor product of the self-dual Haar measure dx i, j. The above integral can we written more explicitly as ( ] An A n X ψ Tr(ɛ X + Tr(Y ɛ ] d X dy, Y A n B m + Y A n X or, after a change of variables, ( ] n An 0 m, ψ : = det A n 2m 0 B m ψ Tr(ɛ A n ( ] An X Y B m + Y A n X X + Tr(Y A ɛ] d X dy. n

7 SMOOTH TRANSFER OF KLOOSTERMAN INTEGRALS 27 The previous form shows that the integral converges and defines a smooth function on GL(n, F M(m m, F. More precisely, the following result is easily obtained. LEMMA 2 If is supported on O n, then n m, ψ : ] is a smooth function of compact support on GL(n, F M(m m, F. Conversely, any smooth function of compact support on GL(n, F M(m m, F is equal to n m, ψ : ] for a suitable function supported on O n. We also introduce the normalized intermediate orbital integrals ( ] ( ] n An 0 m, ψ : := det A n m n An 0 m, ψ :. (5 0 B m 0 B m If is a smooth function of compact support on GL(n, F M(m m, F, we can define its orbital integrals relative to the action (N n N n (N m N m on GL(n, F M(m m, F. They are denoted by (, ψ : x n, x m, where x n is relevant in GL(n, F and x m relevant in M(m m, F. In particular, consider the case of the function ( ] (A n, B m = n An 0 m, ψ : 0 B m and a relevant element x of M((n + m (n + m, F of the form ( xn 0 x =, 0 x m where x n is relevant in GL(n, F and x m is relevant in M(m m, F. We then have the following reduction formula:, ψ : x] =, ψ : x n, x m ]. (6 This formula reduces the computation of the orbital integrals (and the normalization of the measures to the case of an element of the form w n a, where a is a scalar matrix. Then a a ax 2,n, ψ : w n a] = 0 0 a ax n 2,n ax n 2,n 0 a ax n,3 ax n,n ax n,n a ax n,2 ax n,3 ax n,n ax n,n ( i=n ψ x i,n i+2 dx i, j, i=2

8 28 HERVÉ JACQUET where the measures dx i, j are self-dual. We also denote by (, ψ : a, a 2,..., a n the orbital integral (, ψ : a, where a = diag(a, a 2,..., a n. We introduce normalized diagonal orbital integrals as follows. For a A m (F we set Thus σ n (a := (a 2 (a n (a. σ n (a = a n a n 2 2 a n and a is relevant if σ n (a 0. For a relevant, we set (, ψ : a := σ n (a (, ψ : a. (7 We have a reduction formula for these normalized orbital integrals. In a precise way, for S (M(n + m (n + m, F, set ( ] (A n, B m = n An 0 m, ψ :. 0 B m Consider a diagonal matrix ( an 0 a = 0 a m with a n A n (F and a m relevant in A m (F. The normalized orbital integral of on the pair (a n, a m is defined to be It follows from the relation, ψ : a n, a m ] = σ n (a n σ m (a m, ψ : a n, a m ]. that it is equal to (, ψ : a. (det a n m σ n (a n σ (a m = σ n+m (a 3. Weil formula: The split case Our goal is to obtain a simple relation between the orbital integrals of a function and the orbital integrals of its Fourier transform. Our main tool is the Weil formula for the Fourier transform of a character of second order. We state the form of the Weil formula that we are using. We consider the direct sum M(n m, F M(m n, F, the first index being the number of rows. A function on that space is written as ( 0n,n X. Y 0 m,m

9 SMOOTH TRANSFER OF KLOOSTERMAN INTEGRALS 29 Thus the matrix written is a square matrix of size (n+m (n+m with zero diagonal blocks, X M(n m, F, and Y M(m n, F. The Fourier transform of such a function is the function defined by ( ( 0n,n X 0 U ˆ Y 0 m,m := = V 0 ( 0 U V 0 ψ Tr (( 0 X Y 0 ( ] cc0 U du dv V 0 ψ Tr(YU Tr(X V ] du dv. (8 PROPOSITION 2 For A GL(n, F, B GL(m, F, and any Schwartz function on M(n m, F M(m n, F, we have ( 0 X Y 0 ψ (Tr BY AX d X dy = det A m det B n ˆ ( 0 X ψ( Tr B Y A X d X dy. Y 0 Proof Assume first that A and B are unit matrices. Set ( 0 V (U, V = ψ ( Tr(YU du. Y 0 Then the left-hand side of the identity reduces to ( X, X d X. On the other hand, using the (partial Fourier inversion formula, we see that the righthand side is equal to (Y, Y dy. Our assertion then follows. In general, we remark that Tr(BY AX = Tr Y AX B. We change X to X B and Y to Y A in the left-hand side, and we change Y to BY and X to AX in the right-hand side. We see that we are reduced to the case of A =, B = applied to the function ( (X, Y det A m B n 0 X B Y A, 0 the Fourier transform of which is the function (X, Y ˆ ( 0 AX BY 0.

10 30 HERVÉ JACQUET Our assertion follows. We record a simple consequence of this formula. For P M(m n, F and Q M(n m, F, ( 0 X ψ ( Tr(P X + Tr(Y Q + Tr(BY AX d X dy Y 0 = det A m det B n ( 0 X ˆ ψ ( Tr B (Y + PA (X + Q d X dy. Y 0 (9 4. Inversion formula for the orbital integrals For S (M(n n, F, we define the Fourier transform of to be ˆ (X = (Y ψ Tr(XY ] dy. The measure is self-dual. Recall that w n W n is the permutation matrix with unit antidiagonal. We set ˇ (X = ˆ (w n Xw n. Our goal in this section is to obtain a relation between the orbital integrals of and the orbital integrals of ˇ. We first establish such a relation for the intermediate orbital integrals. PROPOSITION 3 Let n, m be two integers. Let S (M((m + n (m + n, F. Then ( ( ] n An 0 m, ψ : ψ Tr B m w m C m w m ] d B m 0 B m ψ Tr(w m Cm w mɛn m A n ɛm n ] ψ Tr(A n w n D n w n ] d A n ( ] = m Cm 0 n ˇ, ψ :. 0 D n The integrals are for B m M(m m, F and A n M(n n, F. Proof We consider the partial Fourier transform (with respect to B m of the normalized intermediate orbital integral of : (A n, C m := n m, ψ : ( An 0 0 B m ] ψ Tr B m C m ] d B m.

11 SMOOTH TRANSFER OF KLOOSTERMAN INTEGRALS 3 If we change C m to w m C m w m and D n to w n D n w n, we see that the identity of the proposition amounts to (A n, C m ψtr C ɛm n ]ψ Tr A n D n ] d A n = det C m n m ɛm n A n ( Dn + w ˆ n Y w m C m w m Xw n w n Y w m C m C m w m Xw n C m ψ Tr ɛm n X Tr Y ɛn m ] d X dy. (0 Changing X to w m Y w n and Y to w n Xw m and noting the relations w n ɛ n m w m = ɛ m n, w m ɛ n m w n = ɛ m n, we see that the right-hand side of the last formula can also be written ( Dn + XC ˆ m Y XC m ψ Tr Y ɛ n m C m Y C Tr ɛm n X] d X dy. ( m From now on, let us write ɛ for ɛn m and ɛ for ɛm n. After a change of variables, we find that is also equal to ( ] det A n m An A n X Y A n B m ψtr(ɛ X + Tr(Y ɛ + Tr(C m Y A n X] d X dy ψ Tr(C m B m ] d B m. After a new change of variables, this becomes ( ] det A n m An X Y B m X + Tr(Y A ψ Tr(ɛ A n ψ Tr(C m B m ] d B m. We now introduce a partial Fourier transform of : ( An V U C m := n ɛ + Tr(C my A n X] d X dy ( ( ( ] An X 0 X cc0 U ψ Tr d X dy d B m. Y B m Y B m V C m By the Weil formula, the previous expression is then ( det C m n An X ψ Tr Cm (Y + ɛ A n Y A n(x + A n ɛ] d X dy. C m

12 32 HERVÉ JACQUET Expanding ψ, we find det C m n ψ Tr(C ( An Y m ɛ A n ɛ] X C m ψ Tr(Cm Y A n X Tr(Cm Y ɛ Tr(C m ɛ X] d X dy. Changing variables once more, we obtain, at last, (A n, C m = det C m n ψ Tr(C ( An XC m C m Y To obtain (0, we must multiply by m ɛ A n C m ɛ] ψ Tr C m Y A n X Tr(Y ɛ Tr(ɛ X ] d X dy ψ Tr(C n ɛ A n and take the Fourier transform of the result with respect to A n. The Fourier transform is evaluated at D n. We indeed obtain (0. The proof shows that for C m GL(m, F, the integral converges as an iterated integral. Our main result in this section is the following inversion formula for diagonal orbital integrals. THEOREM For S (M(n n, F, ( ˇ,ψ : a, a 2,..., a n = (, ψ : p, p 2,, p n ( ψ i=n i= i=n p i a n+ i + i= ɛ] where the multiple integral is only an iterated integral. dp n dp n dp, p i a n i Proof For n = the formula is just the definition of the Fourier transform. Thus we may assume that n > and that our assertion is true for n. We prove it for n. We apply the formula of Proposition 3 to the pair of integers (n,. We obtain, for fixed

13 SMOOTH TRANSFER OF KLOOSTERMAN INTEGRALS 33 a F, ( ( ] n An 0, ψ : ψ p n a ] dp n 0 p n ψ Tr(a ɛ n A n ɛ n ] ψ Tr(A n w n D n w n ] d A n ( ] = a 0 n ˇ, ψ :. 0 D n This formula shows that the function (A n defined by (A n := ψ Tr(a ɛ n A n ɛ n ] ( ] n An 0, ψ : ψ p n a ] dp n, 0 p n which a priori is defined only on GL(n, F, extends in fact to a Schwartz-Bruhat function on M((n (n, F, still denoted by such that ˇ =, where is the Schwartz function on M((n (n, F defined by ( ] (D n = a 0 n ˇ, ψ :. 0 D n The induction hypothesis applied to gives (,ψ : a 2, a 3,..., a n = (, ψ : p, p 2,..., p n On the other hand, with we have ( i=n ψ i= i=n 2 p i a n+ i + i= dp n dp. (2 p i a n i a = diag(a, a 2,..., a m, a = diag(a 2, a 3,..., a n, (,ψ : a 2, a 3,..., a n = σ n (a (, ψ : a = a n σ n (a n N n (F N n (F = σ n (a ( ˇ, ψ : a = ( ˇ, ψ : a, a 2,..., a n. ( ] a 0 ˇ, ψ : 0 t u 2 a θ(u u 2 du du 2 u

14 34 HERVÉ JACQUET Hence we get ( ˇ,ψ : a, a 2,..., a n = (, ψ : p, p 2,..., p n ( i=n ψ i= i=n 2 p i a n+ i + i= dp n dp. (3 p i a n i To complete the proof, we have to compute the orbital integral of the function A n (A n on the matrix p = diag(p, p 2,..., p n. Note that the factor ψ Tr(a ɛ n A n ɛ n ], which appears in the definition of, is constant on the orbits of N n (F N n (F and, in particular, takes the value ψ/(a p n ] on the orbit of p. Observe, furthermore, that if φ is the characteristic function of a compact open set in F, then the function ( ] (A n, p n φ(det A n n An 0, ψ : 0 p n is a smooth function of compact support on GL(n, F F. In particular, ( t ] n u, ψ : xu 2 0 θ(u u 2 ψ p n a ] dp n du du 2 0 p n is an absolutely convergent integral where the order of integration can be reversed. Moreover, for a given p n, σ n (p ( t n u, ψ : p ] u 2 0 θ(u u 2 du du 2 0 p n =, ψ : p, p 2,..., p n ].

15 SMOOTH TRANSFER OF KLOOSTERMAN INTEGRALS 35 It follows that,ψ : p ] ] = ψ σ n (p a p n = ψ ( t n u, ψ : p ] u 2 0 θ(u u 2 ψ p n a ] du du 2 dp n 0 p n ] σ n (p det p a p n ( ( t n u, ψ : p u p n ] = ψ a p n ] = ψ a p n σ n (p ] θ(u u 2 ψ p n a ] du du 2 dp n, ψ : p, p 2,..., p n ]ψ p n a ] dp n, ψ : p, p 2,..., p n ]ψ p n a ] dp n. Upon inserting this result in the right-hand side of (3, we find the identity of the theorem. There is another more elementary formula that comes directly from the Fourier inversion formula. PROPOSITION 4 For S (M(n n, F, n >, the function φ(a =, ψ : w n a] is a smooth function of compact support on F. Furthermore, φ(a = a n2 + ˇ, ψ : ( wn a ] 0 db. 0 b

16 36 HERVÉ JACQUET Proof After a change of variables, we find φ(a = a (n n2 /2 i= a a x 2,n 0 0 a x n 2,n x n 2,n i=n ψ (a x i,n i+2 dx i, j. 0 a x n,3 x n,n x n,n a x n,2 x n,3 x n,n x n,n Since n is bounded above on the support of, the above integral vanishes for a sufficiently large. On the other hand, since the integrand is a smooth function of the x i, j, the integral vanishes for a small enough. We pass to the second assertion. In terms of ˇ we find that φ is equal to ˇ a x,n 0 0 a x n 3,n x n 3,n 0 a x n 2,3 x n 2,n x n 2,n a x n,2 x n,3 x n,n x n,n x n, x n,2 x n,3 x n,n x n,n ( a (n n2/2 ψ a i=n x i,n i+ dx i, j. i= We define a function on GL(n, F by (A n = n ˇ, ψ : ( ] An 0 db. 0 b We remark that if f is a smooth function of compact support on F, then the product f (det A n (A n is a smooth function of compact support on GL(n, F. It easily follows that ˇ, ψ : ( wn a ] 0 db = ( ; ψ : w n a. 0 b

17 SMOOTH TRANSFER OF KLOOSTERMAN INTEGRALS 37 Now (A n = ( ] An A ˇ n X ψ( ɛ n X Y ɛ n d X dy db Y A n b + Y A n X = det A n 2 ( ] An X ˇ ψ( ɛ n A Y b n X Y A n ɛ n d X dy db. Now, ψ : w n a ] = ( i=n (A n ψ x i,n i+ dx i, j, i=2 where A n is the matrix a 0 0 a a x n 3,n. 0 a a x n 2,3 a x n 2,n a a x n,2 a x n,3 a x n,n After a change of variables, we find a ((n 2 (n /2 ( i=n (A n ψ a x i,n i+ dx i, j, where now A n is the matrix a 0 0 a x n 3,n. 0 a x n 2,3 x n 2,n a x n,2 x n,3 x n,n Combining with the previous formula for, we find that its orbital integral on w n a is a ((n 2 (n /2+2(n i=2 ( i=n ˇ (A n ψ a x i,n i+ dx i, j, i=

18 38 HERVÉ JACQUET where A n is now the matrix a x,n 0 0 a x n 3,n x n 3,n 0 a. x n 2,3 x n 2,n x n 2,n a x n,2 x n,3 x n,n x n,n x n, x n,2 x n,3 x n,n x n,n Comparing with the last expression for φ(a, we obtain our assertion. 5. Orbital integrals in H(n n, E/F We let E/F be a quadratic extension of F and denote by H(n n, E/F the space of n n Hermitian matrices, that is, the matrices m such that t m = m. The group N n (E operates on H(n n, E/F by m t umu. Viewed as functions on H(n n, E/F, the functions i are invariants of this action. For i < n, we let O i be the subset of H(n n, E/F defined by i 0. We define an algebraic character θ : N(E F by θ (u = i=n i= u i,i+. Often we write θ (u = θ 0 (uu. We say that an element x (or its orbit is relevant if θ is trivial on the stabilizer of x in N n (E. PROPOSITION 5 In H(n n, E/F the set defined by n = 0, n = 0 contains no relevant orbit. Proof Every s H(n n, E/F has the form s = t nwb with n N n (E, w W n, and b upper triangular. Suppose that det s = 0, and suppose that s is relevant. We have to show that n (s 0. Now, if a column of s with index i < n is zero, then the row with index i is also zero because s is Hermitian, and then s is irrelevant since it is fixed by the group N αi (E. In particular, b, 0. As before, at the cost of replacing s by t n sn, we may assume that s = t nwb, where b has the form b = ( b 0, 0 0 where b is an invertible diagonal matrix in GL(n, E. In particular, the last row of b is zero. We may replace s by the element wbn, and the last row of bn is again zero. Since the rows of wbn with index less than n cannot be zero, w must have the form ( w 0 w = 0 0 with w W n. Then n (wbn = det b det w 0, as claimed.

19 SMOOTH TRANSFER OF KLOOSTERMAN INTEGRALS 39 As before, an element x such that det x = 0 but n (x 0 is in the same orbit as an element of the form ( x Moreover, x is relevant if and only if x is a relevant element in GL(m, E H((m (m, E/F. Now we recall the classification of the relevant orbits in GL(n, E H((n (n, E/F. Every orbit has a unique representative of the form wa with w W, w 2 =, a A m (E, and waw = a (cf. ]. Suppose that α is a simple root such that wα = β where β is positive. For n α N α, define n β N β by t n β wan α = wa. Then t n α wan β = wa. There exists an element n α+β N α+β (i.e., n α+β = if α + β is not a root such that n = n α n β n α+β satisfies t nwan = wa. If wa is relevant, this relation implies θ 0 (n α n α n β n β = 0. If β is not simple, this leads to a contradiction. Thus β is simple. Since w 2 =, we see that, as before, there is a standard Levi subgroup M such that w is the longest element in W m M. The above relation also implies that a is in the center of M and in A(F. We conclude that the set of representatives for the relevant orbits of N m (F N m (F in M(m m, F given in Section 2 is also a set of representatives for the relevant orbits of N m (E in H(m m, E/F. We set θ(uu = ψ(θ 0 (uu. The orbital integrals, ψ, E/F : x] = ( t uxuθ(uu du of a relevant element x are defined as before: the integral is over the quotient of N m (E M by the stabilizer of x. Our goal is to study these orbital integrals in complete analogy with the previous discussion. It is convenient to introduce the intermediate orbital integrals. Let m, n be two integers greater than zero. For S (H((m + n (m + n, E/F, A n S n (F,

20 40 HERVÉ JACQUET B m H(m m, E/F, we set ( ] n An 0 m, E/F, ψ : 0 B m ( ( ( ] n 0 An 0 n X := t X m 0 B m 0 m ψ Tr(ɛ X + Tr( t X ɛ ] d X. (4 Here ɛ and ɛ are as above. Note that ɛ = t ɛ. If X = (x i, j, then the measure d X is the tensor product of the self-dual Haar measures dx i, j. The integral converges and defines a smooth function on S n (F M(m m, F. In particular, we have the following easy result. LEMMA 3 If is supported on O n, then the function n m (, ψ : is a smooth function of compact support on S n (F M(m m, F. Conversely, every smooth function of compact support on that space is of this form for a suitable supported on O n. After a change of variables, we find, keeping in mind that det A n F, n m ( ] An 0, E/F, ψ : 0 B m ( ] = det A n 2m An X F t X B m + t X A n X ψ Tr(ɛ A n X + Tr( t X A n ɛ] d X. We also introduce the normalized intermediate orbital integrals n m ( ] An 0, E/F, ψ : 0 B m := η(det A n m det A n m F n m ( ] An 0, E/F, ψ :. (5 0 B m One can define the orbital integrals of a smooth function of compact support on S n (F H(m m, E/F relative to the action of (N n (E (N m (E on S n (F H(m m, E/F. They are denoted by (, ψ : x n, x m, where x n is relevant in S n (F and x m is relevant in H(m m, E/F. In particular, consider the case of the function ( ] (A n, B m = n An 0 m, ψ, E/F : 0 B m

21 SMOOTH TRANSFER OF KLOOSTERMAN INTEGRALS 4 and a relevant element x of H(m m, E/F of the form ( xn 0 x =, 0 x m where x n is relevant in S n (F and x m is relevant in H(m m, E/F. We then have the reduction formula, E/F, ψ : x] =, E/F, ψ : x n, x m ]. (6 This formula reduces the computation of the orbital integrals (and the normalization of the measures to the case of an element of the form w n a, where a F is a scalar matrix. Then, E/F,ψ : w n a] a a ax 2,n = 0 0 a ax n 2,n ax n 2,n i=2 0 a ax n,3 ax n,n ax n,n a ax n,2 ax n,3 ax n,n ax n,n ( i=n ψ x i,n i+2 dx i, j, where x i,i F, x i, j E, x i, j = x j,i, where the measures dx i, j, i < j, on E and the measures dx i,i on F are self-dual. We also denote by (, E/F, ψ : a, a 2,..., a n the orbital integral (, E/F, ψ : a, where a = diag(a, a 2,..., a n is relevant. We introduce normalized diagonal orbital integrals as follows: (, E/F,ψ : a, a 2,..., a n := η ( σ n (a σ n (a (, E/F, ψ : a, a 2,..., a n. (7 As before, there is a reduction formula for the normalized diagonal orbital integrals. 6. Weil formula: The twisted case As before, our goal is to obtain a simple relation between the orbital integrals of a function and the orbital integrals of its Fourier transform. Again, our main tool is the Weil formula for the Fourier transform of a character of second order. We first recall the one-variable case. Define the Fourier transform of S (E by ˆ (z = (uψ( uz uz du. E

22 42 HERVÉ JACQUET Then, for a F, ˆ (zψ(azz dz = a F η(ac(e/f, ψ ( (zψ zz dz. a We may take this formula as a definition of the constant c(e/f, ψ. We also have ( (zψ(azz dz = a F η(ac(e/f, ψ ˆ (zψ zz dz. a Applying the formula twice, we get the relation c(e/f, ψc(e/f, ψ =. More generally, if is a Schwartz function on the space of column vectors of size n with entries in E, we define its Fourier transform, a function on the space of row vectors of size n, by ˆ (X = (Uψ ZU ZU] du. PROPOSITION 6 For every matrix A S n (F, we have E n ˆ (ZψZ A t Z] d Z = det A F η(det Ac(E/F, ψn E n (Xψ t X A X] d X. Proof If A is a diagonal matrix (with diagonal entries in F, the formula follows at once from the case of n =. In general, we may write A = Ma t M, with M GL(n, E, a diagonal. Then the left-hand side is det M E ˆ (Z M ψza t Z] d Z. Since Z det M ˆ (Z E M is the Fourier transform of X (M X, this is equal to η(det a det a c(e/f, ψ n (M Xψ( t Xa X d X.

23 SMOOTH TRANSFER OF KLOOSTERMAN INTEGRALS 43 After a change of variables, this becomes η(det a det M E det a c(e/f, ψ n (Xψ( t X A X d X. Since det A F = det a F det M E, our assertion follows. On the space H(n n, E/F, we define the Fourier transform by ˆ (X = (Y ψ Tr(XY ] dy, where the measure is self-dual. Thus if Y = (y i, j, then y i,i F, y j,i = y i, j for i < j, and the measure dy is the tensor product of the self-dual Haar measures dy i,i and dy i, j, i < j. Note that Tr(XY is indeed in F. In H((n + m (n + m, E/F we consider the subspace of matrices with zero diagonal n n and m m blocks. The Fourier transform of a function in that space is then defined by ( 0n,n ˆ t ( X 0n,n V = X t ψ Tr(X V + X V ] dv. V 0 m,m 0 m,m This being so, the form of Weil formula that we are using is as given in the next proposition. PROPOSITION 7 Let A, B be Hermitian matrices in GL(n, E and GL(m, E, respectively. Then ( 0n,n ˆ t Z ψ Tr(B Z A t Z ] d Z Z 0 m,m = η(det A m det A m F η(det Bn det B n F c(e/f, ψmn ( 0n,n X t ψ Tr(B t X A X ] d X. X 0 m,m Proof Let us observe that Tr(B Z A t Z is indeed in F. As before, we may reduce the computation to the case where B is the diagonal matrix diag(b, b 2,..., b m. We may regard Z as a matrix Z Z = Z 2, Z m where each Z i is a row of size n. Then the formula follows from the previous formula applied to the matrices b A, b 2 A,..., b m A.

24 44 HERVÉ JACQUET 7. Inversion formula in the Hermitian case As before, we set ˇ (X = ˆ (w m Xw m. We then have the following result. PROPOSITION 8 Let n, m be two integers. Let S ( H((m + n (m + n, E/F. Then we have ( n m ( ] An 0, E/F, ψ : ψ Tr B m w m C m w m ] d B m 0 B m ψ Tr(w m Cm w mɛ A n ɛ] ψ Tr(A n w n D n w n ] d A n ( ] = c(e/f, ψ mn m Cm 0 n ˇ, E/F, ψ :. 0 D n Proof The proof follows step by step the proof of the corresponding result for M((n + m (n + m, F. We consider the partial Fourier transform (with respect to B n of the normalized intermediate orbital integral of : (A n, C m := n m, E/F, ψ : ( An 0 0 B m The critical step is as follows. We get for : ( ] η(det A n m det A n m An X t X B m ψ Tr(ɛ A n ψ Tr(C m B m ] d B m. X + Tr( t X A n ] ψ Tr B m C m ] d B m. ɛ + Tr(C m t X A n X] d X We now introduce a partial Fourier transform of : ( ( ( ( ] An U An X 0 X 0 U t := U C t ψ Tr m X B t m X B t d X d B m. m U C m By the Weil formula, the previous expression is then det C m n η(det C m n c(e/f, ψ mn ( An X t ψ Tr Cm X C (t X + ɛ A n A n(x + A n ɛ] d X. m The rest of the proof is identical to the proof of Proposition 3.

25 SMOOTH TRANSFER OF KLOOSTERMAN INTEGRALS 45 Our main result in this section is the following inversion formula. THEOREM 2 For S (H(n n, E/F, c(e/f,ψ (n(n /2 ( ˇ, ψ, E/F : a, a 2,..., a n = (, ψ, E/F : p, p 2,..., p n ( ψ i=n i= i=n p i a n+ i + where the integral is only an iterated integral. i= dp n dp n dp, p i a n i Proof Again the formula is trivial for n =, so we may assume that n > and that our assertion is true for n. We prove it for n. We apply the formula of Proposition 8 to the pair of integers (n,. We obtain We set ( n, E/F, ψ : ( ] An 0 ψ p n a ] dp n 0 p n ψ Tr(a ɛ n A n ɛ n ] ψ Tr(A n w n D n w n ] d A n ( ] = c(e/f, ψ n a 0 n ˇ, E/F, ψ :. 0 D n (A n := ψ Tr(a ɛ n A n ɛ n ] ( ] n An 0, E/F, ψ : p n a ] dp n. 0 p n Then ˇ (D n = (D n, where ( ] (D n = c(e/f, ψ n a 0 n ˇ, E/F, ψ :. 0 D n The induction hypothesis applied to gives c(e/f,ψ (n (n 2/2 (, E/F, ψ : a 2, a 3,..., a n = (, E/F, ψ : p, p 2,..., p n ( i=n ψ i= i=n 2 p i a n+ i + i= p i a n i dp n dp. (8

26 46 HERVÉ JACQUET The rest of the proof is the same as before once we observe that c(e/f, ψ n c(e/f, ψ (n (n 2/2 = c(e/f, ψ n(n /2. We leave it to the reader to formulate and prove the analogue of Proposition 4 in the present situation. 8. Smooth matching Definition We say that a function S (M(n n, F and a function S (H(n n, E/F have matching orbital integrals for ψ, and we write ψ if for every diagonal matrix a A n with n (a 0, (, ψ; a = (, E/F, ψ : a. From the inversion formula for the orbital integrals, we have at once the following result. PROPOSITION 9 If ψ, then ˇ ψ c(e/f, ψ n(n /2 ˇ, and conversely. Now we can formulate our main result. THEOREM 3 Given S (M(n n, F, there is S (H(n n, E/F with matching orbital integrals for ψ, and conversely. Proof We treat the case of. The case of is similar. The case of n = being trivial, we may assume that n > and that our assertion is proved for n < n. For i < n we can consider smooth functions of compact support on GL(i, F M((n i (n i, F and S i (F H((n i (n i, E/F, respectively, and their orbital integrals for the action of the groups N i (F N i (F N n i (F N n i (F and N i (E N n i (E, respectively. It follows from the induction hypothesis that any function on the first space matches a function on the second space for ψ. Now let O i (resp., O i be the open set of M(n n, F (resp., H(n n, E/F defined by i 0. If is supported on O i, then its normalized intermediate orbital integral i n i, ψ : g i, m n i ]

27 SMOOTH TRANSFER OF KLOOSTERMAN INTEGRALS 47 is a smooth function of compact support i,n i on GL(i, F M((n i (n i, F; it matches a function of compact support i,n i on S i (F H((n i (n i, E/F, which in turn is the normalized intermediate orbital integral of a function supported on O i. Now consider diagonal matrices a i A i, a n i M((n i (n i, F, with n i (a n i 0. Then we have ( ] ai 0, ψ : = ] i,n i, ψ : a i, a n i. 0 a n i There is a similar formula for. Thus, in fact, for supported on O i, i < n. Next, suppose that is such that (, ψ : w n a = 0 ψ. Hence our assertion is true for all a F. Our classification of relevant orbits shows then that all the integrals of over relevant orbits contained in the closed set Q defined by = 2 = = n = 0 vanish. Hence has the same orbital integrals as a function supported on the complement of Q. We may assume that the support of is contained in the complement of Q. Using a partition of unity, we see that we can write = where is supported on O i. Then i on O i and Now let be an arbitrary function. Then i=n i= i, ψ i, where the function i is supported ψ i=n i= i. φ(a := a (n+(n, ψ : w n a] is a smooth function of compact support on F. Let be a function of compact support contained in O n. Then n, ψ : ( ] gn 0 0 b

28 48 HERVÉ JACQUET is an arbitrary smooth function of compact support on GL(n, F F. In particular, we can choose so that ( a ] w φ(a =, ψ : n 0 dp. 0 p Let 2 be the function such that ˇ 2 =. Since matches a function for ψ, it ψ follows from Proposition 9 that 2 2 for a suitable function 2. On the other hand, by Proposition 4 applied to 2, so that (, ψ : w n a = ( 2, ψ : w n a, ( 2, ψ : w n a = 0. ψ By the previous case, it follows that 2 3 for a suitable function 3. Then ψ and our assertion follows. 9. Application to the fundamental lemma Suppose that E/F is unramified, and suppose that the conductor of ψ is O F. Let 0 be the characteristic function of the set of matrices with integral entries in M(m m, F. Similarly, let 0 be the characteristic function of the set of matrices with integral entries in H(m m, E/F. The fundamental lemma asserts that 0 matches 0. We prove this for m = 2 and m = 3. The case where det a = is already known, but the proof presented below is much simpler. First ( 0, ψ : a = ( 0, ψ : a, and likewise for 0. It follows that the difference ω(a := ( 0, ψ : a ( 0, E/F, ψ : a satisfies the functional equation ω(a, a 2,..., a m = ω(p, p 2,..., p m ( i=m ψ i= i=m p i a m+ i + i= dp m dp m dp. p i a m i Moreover, it is supported on the set a a 2 a m. The fundamental lemma asserts that ω = 0. For m = 2 it is easily checked that ( 0, ψ : a, a 2 da 2 (9

29 SMOOTH TRANSFER OF KLOOSTERMAN INTEGRALS 49 is if a = and 0 otherwise, and likewise for ( 0, E/F, ψ : a, a 2 da 2. Thus ω(a, a 2 da 2 = 0. If we set µ(a, b := ω(a, p 2 ψ( p 2 b dp 2, then (9 is equivalent to the relation µ(a, b = µ(b, a ψ ( a b. (20 Moreover, µ(a, b + t = µ(a, b for t a. Since µ(a, 0 = 0, we see that µ(a, b 0 implies b > a. By (20, µ(a, b 0 also implies b > a. Thus µ = 0 or, equivalently, ω = 0, and we are done. For m = 3 we first establish the following lemma. LEMMA 4 If then ω = 0. ω(a, a 2, a 3 da 3 = 0, Proof Indeed, if we set µ(a, a 2, b := ω(a, a 2, a 3 ψ( a 3 b da 3 and ( σ (a, a 2, b := µ(a, a 2, b ψ, a 2 b then (9 is equivalent to σ (a, a 2, b = σ (b, p 2, a ψ(p 2 a 2 dp 2. (2 The condition on the support of ω implies that µ(a, a 2, b + t = µ(a, a 2, b for t a a 2. Under the assumption of the lemma, we thus have µ(a, a 2, b = 0 for b a a 2. Equivalently, σ (a, a 2, b 0 implies a 2 ϖ b a. From (2 we have σ (a, a 2 + t, b = σ (a, a 2, b for t ϖ b a. Thus σ = 0 and ω = 0.

30 50 HERVÉ JACQUET Thus it suffices to prove the relation ( 0, ψ : a, a 2, a 3 da 3 = η(a 2 ( 0, E/F, ψ : a, a 2, a 3 da 3. Now 2 ( 0, ψ : g, a 3 da 3 is det g 2 F times the characteristic function of the set of g M(2 2, F such that g, (0, g, g t (0,. Likewise, 2 ( 0, E/F, ψ : g, a 3 da 3 is det g 2 F times the characteristic function of the set of g H(2 2, E/F such that Thus it suffices to show that g, (0, g. (, ψ : a, a 2 = η(a 2 (, E/F, ψ : a a 2. (22 If a a 2 =, then this relation is equivalent to (, ψ : a, a 2 = η(a (, E/F, ψ : a a 2, where is the characteristic function of GL(2, O F and is the characteristic function of GL(2, O E H(2 2, E/F; in turn, this relation is a special case of the fundamental lemma for m = 2. Now (, ψ : a, a 2 = ψ(x + x 2 dx dx 2 over the set a 2, a x i, x i a, a 2 + a x x 2. If a 2 =, then this is. If a 2 >, then this is zero unless a a 2 =. A similar remark applies to ( 0, E/F, ψ : a, a 2. The relation (22 follows. We are done. 0. Concluding remarks More generally, one can use Propositions 3 and 4 to prove inductively the theorem of density established in 3]; in a precise way, if is such that (, ψ : a = 0 for all diagonal matrices a, then ( ˇ, ψ : a = 0, and (, ψ : g = 0, ( ˇ, ψ : g = 0 for all relevant g. Similarly, one can prove that if ψ, then for every g in the common set of representatives for the two sets of orbits, (, ψ : g = γ (g, ψ (, E/F, ψ : g,

31 SMOOTH TRANSFER OF KLOOSTERMAN INTEGRALS 5 where the constant γ (g, ψ (the transfer factor does not depend on the functions, a result that also follows from 3]. This gives another way to compute the transfer factors. Since the determinant is an invariant of the two actions, it follows from the theorem that for any smooth function of compact support on GL(m, F there is a function of compact support on S m (F such that ψ. Except for the last section, the previous discussion applies directly to the extension C/R. Suppose that E/F is unramified, and suppose that the conductor of ψ is O F. Then c(e/f, ψ =. If E/F is a quadratic extension of global fields and ψ is a nontrivial character of F A /F, then the product of the constants c(e v /F v, ψ v over all places v of F inert in E is. Likewise, for g relevant in GL(n, F S n (F the product of the local transfer factors is. References ] D. BUMP, S. FRIEDBERG, and D. GOLDFELD, Poincaré series and Kloosterman sums in The Selberg Trace Formula and Related Topics (Brunswick, Maine, 984, Contemp. Math. 53, Amer. Math. Soc., Providence, 986, MR 87j: ] H. JACQUET, The continuous spectrum of the relative trace formula for GL(3 over a quadratic extension, Israel J. Math. 89 (995, 59. MR 96a: ], A theorem of density for Kloosterman integrals, Asian J. Math. 2 (998, MR 200e:054 22, 50, 5 4] H. JACQUET and Y. YE, Relative Kloosterman integrals for GL(3, Bull. Soc. Math. France 20 (992, MR 94c: ], Distinguished representations and quadratic base change for GL(3, Trans. Amer. Math. Soc. 348 (996, MR 96h: ], Germs of Kloosterman integrals for GL(3, Trans. Amer. Math. Soc. 35 (999, MR 99j:053 7], Kloosterman integrals over a quadratic extension, to appear in Ann. of Math. ( ] E. LAPID and J. ROGAWSKI, Periods of Eisenstein series: The Galois case, Duke Math. J. 20 (2003, ] B. C. NGÔ, Faisceaux pervers, homomorphisme de changement de base et lemme fondamental de Jacquet et Ye, Ann. Sci. École Norm. Sup. (4 32 (999, MR 200g: ], Le lemme fondamental de Jacquet et Ye en caractéristique positive, Duke Math. J. 96 (999, MR 2000f: ] T. A. SPRINGER, Some results on algebraic groups with involutions in Algebraic Groups and Related Topics (Kyoto/Nagoya, 983, Adv. Stud. Pure Math. 6, North-Holland, Amsterdam, 985, MR 86m:

32 52 HERVÉ JACQUET 2] G. STEVENS, Poincaré series on GL(r and Kloostermann sums, Math. Ann. 277 (987, MR 88m: ] J. L. WALDSPURGER, Le lemme fondamental implique le transfert, Compositio Math. 05 (997, MR 98h: ] Y. YE, The fundamental lemma of a relative trace formula for GL(3, Compositio Math. 89 (993, MR 95b: ], An integral transform and its applications, Math. Ann. 30 (994, MR 95j:045 Department of Mathematics, Columbia University, Mail Code 4408, New York, New York 0027, USA; hj@math.columbia.edu