ON MICROFUNCTIONS AT THE BOUNDARY ALONG CR MANIFOLDS

ON MICROFUNCTIONS AT THE BOUNDARY ALONG CR MANIFOLDS Andrea D Agnolo Giuseppe Zampieri Abstract. Let X be a complex analytic manifold, M X a C 2 submanifold, Ω M an open set with C 2 boundary S = Ω. Denote by µ M (O X ) (resp. µ Ω (O X )) the microlocalization along M (resp. Ω) of the sheaf O X of holomorphic functions. In the literature (cf [A-G], [B-C-T], [K-S 1,2]) one encounters two classical results concerning the vanishing of the cohomology groups H j µ M (O X ) p for p T M X. The first one gives the vanishing outside a range of indices j whose length is equal to s 0 (M, p) (where s +,,0 (M, p) denotes the number of positive, negative and null eigenvalues for the microlocal Levi form L M (p)). The second, sharpest result gives the concentration in a single degree, provided that the difference s (M, p ) γ(m, p ) is locally constant for p TM X near p (where γ(m, p) = dimc (TM X it M X) z for z the base point of p). The first result was restated for the complex µ Ω (O X ) in [D A-Z 2], in the case codim M S = 1. We extend it here to any codimension and moreover we also restate for µ Ω (O X ) the second vanishing theorem. We also point out that the principle of our proof, related to a criterion for constancy of sheaves due to [K-S 1], is a quite new one. 1. Notations Let X be a complex analytic manifold and M X a C 2 submanifold. One denotes by π : T X X and π : TM X M the cotangent bundle to X and the conormal bundle to M in X respectively. Let T X be the cotangent bundle with the zero section removed, and let ρ : M X T X T M be the projection associated to the embedding M X. For a subset A X one defines the strict normal cone of A in X by N X (A) := T X \ C(X \ A, A) where C(, ) denotes the normal Whitney cone (cf [K-S 1]). Let z M, p ( T M X) z. We put Tz C M = T z M it z M λ M (p) = T p TM X λ 0 (p) = T p (π 1 π(p)), ν(p) = the complex Euler radial field at p, and we set γ(m, p) = dim C (TM X it M X) z. If no confusion may arises, we will sometimes drop the indices z or p in the above notations. Let φ be a C 2 function in X with φ M 0 and p = (z ; dφ(z )). In a local system of coordinate (z) at z in X we define L φ (z ) as the Hermitian form with Appeared in: Compositio Math. 105 (1997), no. 2, 141 146. 1

2 A. D AGNOLO G. ZAMPIERI matrix ( z i z j φ) ij. Its restriction L M (p) to T C z M does not depend on the choice of φ and is called the Levi form of M at p. Let s +,,0 (M, p) denote the number of positive, negative and null eigenvalues of L M (p). One denotes by D b (X) the derived category of the category of bounded complexes of sheaves of C-vector spaces and by D b (X; p) the localization of D b (X) at p T X, i.e. the localization of D b (X) with respect to the null system {F D b (X); p / SS(F )} (here SS(F ) denotes the micro-support in the sense of [K-S 2], a closed conic involutive subset of T X). Remark 1.1. We recall that a complex F which verifies SS(F ) T M X in a neighborhood of p T M X is microlocally isomorphic (i.e. isomorphic in Db (X; p)) to a constant sheaf on M. This criterion, stated in [K-S 1] for a C 2 manifold M, extends easily to C 1 manifolds (cf [D A-Z 1]). Let O X be the sheaf of germs of holomorphic functions on X and C A (A X locally closed) the sheaf which is zero in X \ A and the constant sheaf with fiber C in A. We will consider the complex µ A (O X ) := µhom(c A, O X ) of microfunctions along A (where µhom(, ) is the bifunctor of [K-S 1]). Of particular interest are the complexes µ M (O X ) and µ Ω (O X ) for Ω being an open subset of the manifold M (cf [S]). 2. Statement of the results Let X be a complex analytic manifold of dimension n, M X a C 2 submanifold of codimension l, Ω M an open set with C 2 boundary S = Ω, and set r = codim M S (we assume Ω locally on one side of S for r = 1). Define: d M (p) = codim X M + s (M, p) γ(m, p), c M (p) = n s + (M, p) + γ(m, p). Let us recall the following classical results concerning the cohomology of µ M (O X ): Theorem A. ( [A-G], [B-C-T], [K-S 1]) Assume dim R (ν(p) λ M (p)) = 1. H j µ M (O X ) p = 0 for j / [d M (p), c M (p)]. Theorem B. ( [H], [K-S 1]) Assume dim R (ν(p) λ M (p)) = 1 and s (M, p ) γ(m, p ) is constant for p TM X close to p. H j µ M (O X ) p = 0 for j d M (p). Dealing with µ Ω (O X ), one knows that: Theorem C. ( [D A-Z 2]) Assume codim M S = 1 and dim R (ν(p) λ M (p)) = 1. H j µ Ω (O X ) p = 0 for j / [d M (p), c M (p)]. The aim of the present note is, on the one hand, to extend Theorem C to the case of any codimension for S in M, and, on the other hand, to state the analogue of Theorem B for the complex µ Ω (O X ). We point out that the method of our proof, based on the criterion of [K-S 1, Proposition 6.2.2] (with its C 1 -variant of [D A-Z 1]), is a quite new one. Our results, valid for any r = codim M S, go as follows.

Theorem 2.1. Assume ON MICROFUNCTIONS AT THE BOUNDARY 3 (2.1) dim R (ν(p) λ S (p)) = 1. (2.2) H j µ Ω (O X ) p = 0 for j / [d M (p), c M (p) + r 1]. Theorem 2.2. Assume (2.1) and moreover s (M, p ) γ(m, p ) is constant for p Ω M TM X near p, (2.3) s (S, p ) γ(s, p ) is constant for p TS X ρ 1 (N M (Ω) a ) near p, s (M, p) γ(m, p) = s (S, p) γ(s, p). (2.4) H j µ Ω (O X ) p = 0 for j / [d M (p), d M (p) + r 1]. Remark 2.3. We notice that the sets appearing in (2.3) are very natural in this context, one has in fact: M X SS(C Ω ) = Ω M T M X, T S X SS(C Ω) = T S X ρ 1 (N M (Ω) a ). 3. Proofs of the results Proof of Theorem 2.1. We set Ω = M \ Ω and use the distinguished triangle (3.1) µ S (O X ) µ M (O X ) µ Ω (O X ) µ Ω (O X ) +1. Recall that (cf [D A-Z 4]): (3.2) c M (p) c S (p) c M (p) + r, d M (p) d S (p) d M (p) + r. The vanishing of (2.2) for j > c M (p) + r 1 then follows by applying Theorem A to M and S. The vanishing of (2.2) for j < d M (p) is immediate for d S (p) > d M (p) due to Theorem A and (3.1). When d S (p) = d M (p) it remains to be proven that (3.3) H d M (p) µ S (O X ) p H d M (p) µ M (O X ) p is injective. To this end we perform a contact transformation χ near p which interchanges (setting q = χ(p)) (3.4) M X T f M X codim M = 1, s ( M, q) = 0 T S X T e S X codim S = 1 (cf [D A-Z 3]). Let M + and S + be the closed half spaces with boundary M and S and inner conormal q. We have

4 A. D AGNOLO G. ZAMPIERI Proposition 3.1. Let d S = d M. in the above situation (3.5) { s ( S, q) = 0, S + M +. Proof. Quantizing χ by a kernel K Ob(D b (X X)) we get by [K-S 1, Proposition 11.2.8]: { φk (C M ) = C fm +[d M (p) 1] in D b (X; q), φ K (C S ) = C es +[d S (p) s ( S, q) 1] in D b (X; q). Moreover the natural morphism C M C S is transformed via φ K to a non null morphism C fm +[d M (p) 1] C es +[d S (p) s ( S, q) 1]. Thus Hom Db (X;q)(C fm +[d M (p) 1], C es +[d S (p) s ( S, q) 1]) = = H 0 (RΓ fm +(C es +) y [d S (p) d M (p) s ( S, q)]) 0, where y = π(q). Since we are assuming d M (p) = d S (p), (3.5) follows. q.e.d. End of the proof of Theorem 2.1. From the proof of Proposition 3.1 it follows that φ K transforms the morphism (3.3) in (3.6) H 1 e S +(O X ) y H 1 f M +(O X ) y, where y = π(q), which is clearly injective. The proof of Theorem 2.1 is now complete q.e.d. Proof of Theorem 2.2. From now on we will drop p in our notations, due to the constancy assumptions (2.3). If r > 1 one has Ω = M and N M (Ω) a = T M. Thus, by (2.3), we enter the hypotheses of Theorem B for both M and S. The claim follows in this case from (3.1), (3.3) and from the inequalities (3.2). We may then assume r = 1. The problem in this case is that (2.3) holds only along SS(C Ω ). Let χ : T X T X be a contact transformation from a neighborhood of p to a neighborhood of q = χ(p), such that: M X T f M X codim M = 1 T S X T e S X codim S = 1, s ( S, q ) 0. Notice that, for y = π(q), T y M = Ty S. Quantizing χ by a kernel K, we thus have that either φ K (C Ω ) or φ K (C Ω ) is a simple sheaf along the conormal bundle to a C 1 submanifold Y X. Since d M = d S 1, then s ( M, q) = 0, M + S + and φ K (C Ω ) = C Y [d M 1]. Denoting by W the open domain with boundary Y and exterior conormal q, we have by [Z] that W is pseudoconvex at y, and one concludes since χ µ Ω (O X ) q [ d M ] = H 1 X\W (O X) y, q.e.d.

ON MICROFUNCTIONS AT THE BOUNDARY 5 References [A-G] A. Andreotti, H. Grauert, Théorèmes de finitude pour la cohomologie des espaces complexes, Bull. Soc. Math. France 90 (1962), 193 259. [B-C-T] M. S. Baouendi, C. H. Chang, F. Treves, Microlocal hypo-analyticity and extension of C-R-functions, J. Diff. Geometry 18 (1983), 331-391. [D A-Z 1] A. D Agnolo, G. Zampieri, A propagation theorem for a class of sheaves of microfunctions, Rend. Mat. Acc. Lincei, s. 9 1 (1990), 53 58. [D A-Z 2] A. D Agnolo, G. Zampieri, Vanishing theorem for sheaves of microfunctions at the boundary on CR-manifolds, Commun. in P.D.E. 17 (1992), no. 5& 6, 989 999. [D A-Z 3] A. D Agnolo, G. Zampieri, Generalized Levi forms for microdifferential systems (M. Kashiwara, T. Monteiro Fernandes, P. Schapira, eds.), D-modules and microlocal geometry, Walter de Gruyter & Co., 1992. [D A-Z 4] A. D Agnolo, G. Zampieri, Levi s forms of higher codimensional submanifolds, Rend. Mat. Acc. Lincei, s. 9, 2 (1991), 29 33. [H] L. Hörmander, An introduction to complex analysis in several complex variables, Van Norstrand. Princeton (1966). [K-S 1] M. Kashiwara, P. Schapira, Microlocal study of sheaves, Astérisque 128 (1985). [K-S 2] M. Kashiwara, P. Schapira, Sheaves on manifolds, Springer-Verlag 292 (1990). [S] P. Schapira, Front d onde analytique au bord II, Sem. E.D.P. Ecole Polyt., Exp. XIII (1986). [S-K-K] M. Sato, T. Kawai, M. Kashiwara, Hyperfunctions and pseudo-differential equations, Lecture Notes in Math., Springer-Verlag 287 (1973), 265 529. [Z] G. Zampieri, The Andreotti-Grauert vanishing theorem for polyhedrons of C n, preprint.