STABLE HYPERSURFACES WITH CONSTANT SCALAR CURVATURE

PROCEEDINGS OF THE AERICAN ATHEATICAL SOCIETY Volume 38, Number 9, September 00, Pages 330 33 S 000-9939(0)0388-8 Article electronically published on April, 00 STABLE HYPERSURFACES ITH CONSTANT SCALAR CURVATURE HILÁRIO ALENCAR, ALCY SANTOS, AND DETANG ZHOU (Communicated by Chuu-Lian Terng) Abstract. e obtain some nonexistence results for complete noncompact stable hypersurfaces with nonnegative constant scalar curvature in Euclidean spaces. As a special case we prove that there is no complete noncompact strongly stable hypersurface in R 4 with zero scalar curvature S, nonzero Gauss-Kronecker curvature and finite total curvature (i.e. A 3 < + ).. Introduction In this paper we study the complete noncompact stable hypersurfaces with constant scalar curvature in Euclidean spaces. It has been proved by Cheng and Yau [CY] that any complete noncompact hypersurfaces in the Euclidean space with constant scalar curvature and nonnegative sectional curvature must be a generalized cylinder. Note that the assumption of nonnegative sectional curvature is a strong condition for hypersurfaces in the Euclidean space with zero scalar curvature since the hypersurface has to be flat in this case. Let n be a complete orientable Riemannian manifold and let x : n R n+ be an isometric immersion into the Euclidean space R n+ with constant scalar curvature. e can choose a global unit normal vector field N, and the Riemannian connections and of and R n+, respectively, are related by X Y = X Y + A(X),Y N, where A is the second fundamental form of the immersion, defined by A(X) = X N. Let λ,..., λ n be the eigenvalues of A. The r-mean curvature of the immersion in a point p is defined by H r = ( n ( n r, r) r)s i <...<i r λ i...λ ir = where S r is the r-symmetric function of the λ,..., λ n, H 0 =andh r =0,for all r n +. For r =,H = H is the mean curvature of the immersion, in the case r =,H is the normalized scalar curvature, and for r = n, H n is the Gauss-Kronecker curvature. Received by the editors September 0, 009 and, in revised form, December 5, 009. 00 athematics Subject Classification. Primary 53C4. The authors were partially supported by CNPq and FAPERJ, Brazil. 330 c 00 American athematical Society Reverts to public domain 8 years from publication

330 HILÁRIO ALENCAR, ALCY SANTOS, AND DETANG ZHOU It is well-known that hypersurfaces with constant scalar curvature are critical points for a geometric variational problem, namely, that associated to the functional () A () = S d, under compactly supported variations that preserve volume. Let be a hypersurface in the Euclidean space with constant scalar curvature. Following [AdCE], when the scalar curvature is zero, we say that a regular domain D is stable if the critical point is such that ( d A dt ) t=0 0 for all variations with compact support in D, and when the scalar curvature is nonzero, we say that a regular domain D is strongly stable if the critical point is such that ( d A dt ) t=0 0, for all variations with compact support in D. It is natural to study the global properties of hypersurfaces in the Euclidean space with constant scalar curvature. For example, we have the following open question (see 4.3 in [AdCE]). Question.. Is there any stable complete hypersurface in R 4 with zero scalar curvature and nonzero Gauss-Kronecker curvature? e have a partial answer to Question.. Theorem A (see Theorem 3.). There is no complete noncompact stable hypersurface in R n+ with zero scalar curvature S and 3-mean curvature S 3 0 satisfying () lim R + where B R is the geodesic ball in. hen S =0,S = A we have B R S 3 R =0, Corollary B. There is no complete noncompact stable hypersurface in R 4 with zero scalar curvature S, nonzero Gauss-Kronecker curvature and finite total curvature (i.e. A 3 < + ). e remark that Shen and Zhu (see [SZ]) proved that a complete stable minimal n-dimensional hypersurface in R n+ with finite total curvature must be a hyperplane. The above corollary can be seen as a similar result in dimension 3 for hypersurfaces with zero scalar curvature. e also prove the following result for hypersurfaces with positive constant scalar curvature in Euclidean space. Theorem C (see Theorem 3.). There is no complete immersed strongly stable hypersurface n R n+, n 3, with positive constant scalar curvature and polynomial growth of -volume; that is B lim R S d R R n <, where B R is a geodesic ball of radius R of n. As a consequence of the properties of a graph with constant scalar curvature, we have the following corollary: Corollary D (see Corollary 4.). Any entire graph on R n with nonnegative constant scalar curvature must have zero scalar curvature.

STABLE HYPERSURFACES ITH CONSTANT SCALAR CURVATURE 3303 This can be compared with a result of Chern [Ch], which says any entire graph on R n with constant mean curvature must be minimal. It has been known by a result of X. Cheng in [Che] (see also [ENR]) that any complete noncompact stable hypersurface in R n+ with constant mean curvature must be minimal if n<5. It is natural to ask that any complete noncompact stable hypersurface in R n+ with nonnegative constant scalar curvature must have zero scalar curvature. It should be remarked that Chern [Ch] proved that there is no entire graph on R n with Ricci curvature less than a negative constant. e don t know whether thereexistsanentiregraphonr n with constant negative scalar curvature. The rest of this paper is organized as follows: we include some results and definitions which will be used in the proof of our theorems in Section. The proof of main results are given in Section 3, and Section 4 is an appendix in which we prove some stability properties for graphs with constant scalar curvature in the Eucildean space.. Some stability and index properties for hypersurfaces with S = const. e introduce the r th Newton transformation, P r : T p T p, which is defined inductively by P 0 = I, P r = S r I A P r,r. The following formulas are useful in the proof (see [Re], Lemma.): (3) trace(p r ) = (n r)s r, (4) trace(a P r ) = (r +)S r+, (5) trace(a P r ) = S S r+ (r +)S r+. From [AdCC] we have the second variation formula for hypersurfaces in a space form of constant curvature c, Q n+ c, with constant -mean curvature: (6) d A dt t=0 = P ( f), f d (S S 3S 3 +c(n )S )f d, f Cc (D). D D Definition.. hen S =0andc =0, is stable if and only if (7) P ( f), f d 3 S 3 f d, for any f Cc (). One can see that if P 0, then S 3 =0and is stable. hen S = const. 0, is stable if and only if P ( f), f d (S S 3S 3 + c(n )S )f d, D D for all f Cc () and fd =0. esaythat is strongly stable if and only if the above inequality holds for all f Cc (). Similar to minimal hypersurface we can also define the index I for hypersurfaces with constant scalar curvature. Given a relatively compact domain Ω, we denote by Ind (Ω) the number of linearly independent normal deformations with support on Ω that decrease A.Theindex of the immersion is defined as (8) Ind () :=sup{ind (Ω) Ω, Ω relatively compact}.

3304 HILÁRIO ALENCAR, ALCY SANTOS, AND DETANG ZHOU is strongly stable if Ind () = 0. The following result has been shown in [El]. Lemma.. Let n Q n+ c be a noncompact hypersurface with S = const. > 0. If has finite index, then there exists a compact set K such that \ K is strongly stable. For hypersurfaces with constant mean curvature, do Carmo and Zhou [dcz] proved that Theorem.. Let x : n n+ be an isometric immersion with constant mean curvature H. Assume has subexponential volume growth and finite index. Then there exists a constant R 0 such that H Ric \BR0 (N), where N is a smooth normal vector field along and Ric(N) is the Ricci curvature of in the normal vector N. The technique in [dcz] was generalized by Elbert [El] to prove the following result: Theorem.. Let x : n Q(c) n+ be an isometric immersion with S = const. > 0. Assume that Ind < and that the -volume of is infinite and has polynomial growth. Then c is negative and S c. In particular, this theorem implies that when c = 0 the hypersurfaces in the above theorem must have nonpositive scalar curvature. 3. Proof of the theorems hen S =0weknowthat S = A.Thus,ifS 3 0,wehavethat A > 0. Hence S 0 and we can choose an orientation such that P is semi-positive definite. Since P A =trace(a P ) = 3S 3, then, when c =0, is stable if (9) P ( f), f d P A f d, for any f Cc (). hen S = 0, we have the following inequality, which is essentially due to Cheng and Yau [CY] (see also Lemma 4. in [AdCC] or Lemma 3. in [Li]): (0) A S 0. In the following lemma, we characterize the equality case in some special case. Lemma 3.. Let n (n 3) be a nonflat connected immersed -minimal hypersurface in R n+.if A = S holds on all nonvanishing points of A in, then each component of with A 0must be a cylinder over a curve.

STABLE HYPERSURFACES ITH CONSTANT SCALAR CURVATURE 3305 Proof. To prove our lemma we recall the computations in [SSY]. Choose a frame at p so that the second fundamental form is diagonalized. e have A = i h ii and h ijk A i,j,k =[( i,j h ij)( s,t,k h stk) k ( i,j h ij h ijk ) ]( i,j h ij) () = (h ij h stk h st h ijk ) A i,j,k,s,t = ii h stk h st h iik ) i,k,s,t(h + h ss( s k i j h ijk) = ii h ssk h ss h iik ) i,k,s(h A + h iij + i j A h ijk. i j,j k,i k The right hand side of the above equation is nonnegative and zero if and only if all terms on the right hand side of the last equation are zero. Suppose x : R n+ is the -minimal immersion. Since is not a hyperplane, then A is a nonnegative continuous function which does not vanish identically. Let p be such a point such that A (p) > 0. Then A > 0 in a connected open set U containing p. The equality in (0) implies h jji =0, for all j i, h ijk =0, for all j i, j k, k i, h ii h ssk = h ss h iik, for all i, s, k. So we have h jij =0, for all j i, and from the last equation we claim at most one i such that h iii 0. Otherwise, without the loss of generality we assume h 0 and h 0;wehaveh h k = h h k for all k. This implies h = h =0 by choosing k =,. Using the third formula again we have h jj h = h h jj for j =3,...,n. Hence h jj =0forallj =3,...,n, which contradicts A 0. e now assume h 0; by continuity we can also assume h 0. Fromthe last line of the above equation, we have h h ss = h ss h for s. Hence h ss =0 for all s. This implies that is a cylinder over a curve. e are now ready to prove Theorem 3.. There is no complete noncompact stable hypersurface in R n+ with S =0and S 3 0satisfying B lim R S 3 R + R =0. Proof. Assume for the sake of contradiction that there is such a hypersurface. From Lemma 3.7 in [AdCC], we have () L S = A S +3S S 3.

3306 HILÁRIO ALENCAR, ALCY SANTOS, AND DETANG ZHOU Since for any φ Cc (), P ( (φs )), (φs ) d = P (( φ)s + φ S ), ( φ)s + φ S d = P ( φ), φ Sd + P ( φ), S φs d + φ P ( S ), S d, using () we have φ S ( A S )d = (L S 3S S 3 )φ S d = P ( S ), (φ S ) d 3S 3 φ S d = φ P ( S ), S d P ( φ), S φs d 3S 3 φ Sd = P ( (φs )), (φs ) d + P ( φ), (φ) Sd 3S 3 φ Sd P ( φ), φ Sd (n ) φ Sd, 3 for any φ Cc (). Here we have used the stability inequality (7) in the seventh line and we use the following consequence of (3) in the last inequality: (3) (n )S φ P ( φ), φ. e can choose φ as R r(x) R, on B R \ B R ; φ(x) =, on B R ; 0, on \ B R. Thus from the choice of φ we have S ( A S ) 0. Therefore the elipticity of L implies L S =3S S 3. From Lemma 3., mustbeacylinderoveracurve, which contradicts S 3 0. The proof is complete. The following lemma is of some independent interest, and we include it here since its second part is useful in the proof of Theorem 3.. Lemma 3.. Let be a complete immersed hypersurface in Q n+ c constant scalar curvature S > n(n ) c and S 0. with positive

STABLE HYPERSURFACES ITH CONSTANT SCALAR CURVATURE 3307 ) If is strongly stable outside a compact subset, then either has finite -volume or lim R + R S =+. B R ) If is strongly stable, then lim R + R S =+. B R In particular has infinite -volume. Proof. e can assume that there exists a geodesic ball B R0 such that \B R0 is strongly stable. That is, (4) (S S 3S 3 + c(n )S )f d P ( f), f d, for all f Cc ( \ B R0 ). Now, since S > 0, we have (see [AdCR], p. 39) H H H 3 and H H /. n(n ) n(n (n ) By using S = nh, S = H and S 3 = H 3, we obtain that 6 (n ) S S 3S 3 ; n that is, (n ) (5) 3S 3 S S. n e also have that ( ) / S n S, n(n ) which implies ( ) / n (6) S S /. n By using inequality (5) in (4), we obtain that ( S S n ) n S S + c(n )S f d P ( f), f d; that is, ( ) n(n )c S + S f d n P ( f), f d. By using (3), we obtain that (n ) S f d P ( f), f d. Therefore, there exists a constant C>0such that (7) S f d C S f d.

3308 HILÁRIO ALENCAR, ALCY SANTOS, AND DETANG ZHOU ) hen is strongly stable outside B R0 we can choose f as r(x) R 0, on B R0 + \ B R0 ;, on B R+R0 + \ B R0 +; f(x) = R+R 0 + r(x) R, on B R+R0 + \ B R+R0 +; 0, on \ B R+R0 +, where r(x) is the distance function to a fixed point. Then R S d + S d C B R+R0 +\B R+R0 + B R0 +\B R0 If the -volume is infinite, we can choose R large enough such that C S d > S d; B R+R0 +\B R0 + B R0 +\B R0 hence lim R + R S d =+. B R+R0 +\B R+R0 + B R+R0 +\B R0 + ) hen is strongly stable we can choose a simpler test function f as, on B R ; R r(x) f(x) = R, on B R \ B R ; 0, on \ B R, which implies that when S 0, The proof is complete. lim R + R S d =+. B R S d. Theorem 3.. There is no complete immersed strongly stable hypersurface n R n+, n 3, with positive constant scalar curvature and polynomial growth of -volume; that is, lim R B R S d R n <, where B R is a geodesic ball of radius R of n. Proof. Suppose that is a completely immersed strongly stable hypersurface n R n+, n 3, with positive constant scalar curvature. From Theorem. it suffices to show that the -volume S d is infinite, which is part () of Lemma 3.. 4. Graphs with S = const. in Euclidean space In this section we include some stability properties and estimates for entire graphs on R n which may be known to experts but may not be easy to find a reference for. Using these facts we give the proof of Corollary 4.. Let n be a hypersurface of R n+ given by a graph of a function u : R n R of class C (R n ). For such hypersurfaces we have:

STABLE HYPERSURFACES ITH CONSTANT SCALAR CURVATURE 3309 Proposition 4.. Let n be a graph of a function u : R n R of class C (R n ). Then () If S =0and S does not change sign on, then n is a stable hypersurface. () If has S = C>0, then n is strongly stable. Proof. Consider f : R a C function with compact support and let = + u. In order to calculate P ( f), f, writeg = f.thus P ( f), f = P ( ( g )), ( g ) = P (g + g ),g ( )+ g = gp ( )+ P ( g),g ( )+ g = g P ( ), + g P ( ), g + g P ( g), + P ( g), g. By using the fact that P is selfadjoint, we have (8) P ( f), f = g P ( ), + g P ( On the other hand, if {e,..., e n } is a geodesic frame along, ), g + P ( g), g. div(fgp ( )) = = = n ei (fgp ( )),e i i= n fg i P ( )+f igp ( )+fg e i (P ( )),e i i= n {fg i P ( ),e i + f i g P ( ),e i + fg ei (P ( )),e i }. i= Since f = g,weget that is, ( ) f i = g i + g ; i ( gf i = gg i + g ) = fg i + g ( ). i i

330 HILÁRIO ALENCAR, ALCY SANTOS, AND DETANG ZHOU Hence, div(fgp ( )) n = {fg i P ( ( ) ),e i +(fg i + g i= n = {fg i P ( ( ) ),e i + g i= ) P ( i ),e i } + fgl ( ) P ( i ),e i } + fgl ( ) =f P ( ), g + g P ( ), ( ) + fgl ( ) = g,p ( g) + g P ( ), ( ) + f L ( ). Thus, (9) g,p ( g) = div(fgp ( )) g P ( ), ( ) f L ( ). Now, by using (9) in equation (8), we get P ( f), f = div(fgp ( )) f L ( )+ P ( g), g. Now, the divergence theorem implies that P ( f), f d = f L ( )d + P ( g), g d. Choose the orientation of in such way that S 0. Since S A =S 0, we obtain that S A. Thus, P ( g), g = S g A g, g (S A ) g 0, which implies that (0) P ( f), f d f L ( )d. hen S is constant, we will use the following formula proved by Reilly (see [Re], Proposition C): L ( )=L ( N,e n+ )= (S S 3S 3 ) N,e n+ = (S S 3S 3 ), where N is the normal vector of and e n+ =(0,..., 0, ±), according to our choice of the orientation of. Thus, P ( f), f d (S S 3S 3 )f d for all functions f with compact support. Hence is stable if S = 0 and strongly stable in the case S 0. Remark 4.. e would like to remark that the operator L need not be elliptic in the above proof. Proposition 4.. Let n be a graph of a function u : R n R of class C (R n ), with S 0. LetB R be a geodesic ball of radius R in. Then S d C(n) B θr θ Rn,

STABLE HYPERSURFACES ITH CONSTANT SCALAR CURVATURE 33 where C(n) and θ are constants, with 0 <θ<. polynomial growth. In particular, S d has Proof. Let f : R be a function in C0 (), that is, a smooth function with compact support. Observe that ( div f u ) ( ) u = fdiv + f, u, where = ( ) u + u. By using the fact that S is given by S =div,we have that ( ) u () fs d = fdiv d = f, u d. Now, choose a family of geodesic balls B R that exhausts. Fixθ, with0<θ <, and let f : R be a continuous function that is one on B θr, zero outside B R and linear on B R \ B θr. Therefore, from equation () we obtain S d fs d u B θr B R, f d. By using Cauchy-Schwarz inequality and the fact that u S d B θr f d B R B R, we get that B r \B θr ( θ)r d ( θ)r vol(b R). e observe that since is a graph, if Ω R = {(x,...,x n+ ) R n+ R x n+ R; x + + x n R}, then vol(b R ) dx...dx n+ = C(n)R n+. Ω R Hence, S d B θr ( θ)r vol(b R)= C(n) θ Rn. e have the following corollary of Theorem 3.. Corollary 4.. Any entire graph on R n with nonnegative constant scalar curvature must have zero scalar curvature. Proof. Suppose for sake of contradiction that there exists an entire graph with S = const. > 0. Such a graph is strongly stable; and if S > 0, we get that S = A +S > 0, we obtain that S does not change sign, and we can choose the orientation in such a way that S > 0. Thus the graph has polynomial growth of the -volume. Thus we have a contradiction with Theorem 3.. Therefore it follows that S =0.

33 HILÁRIO ALENCAR, ALCY SANTOS, AND DETANG ZHOU References [AdCC] Alencar, H., do Carmo,., Colares, A.G., Stable hypersurfaces with constant scalar curvature, ath. Z. 3 (993), 7 3. R7674 (94d:53080) [AdCE] Alencar, H., do Carmo,., Elbert,.F., Stablity of hypersurfaces with vanishing r-constant curvatures in Euclidean spaces, J. Reine Angew. ath. 554 (003), 0 6. R9573 (003k:5306) [AdCR] Alencar, H., do Carmo,., Rosenberg, H., On the first eigenvalue of the linearized operator of the rth mean curvature of a hypersurface. Annals of Global Analysis and Geometry, no. 4 (993), 387 395. R4697 (94m:53086) [ASZ] Alencar, H., Santos,., Zhou, D., Curvature integral estimates for complete hypersurfaces, arxiv:0903.035. [Che] Cheng, X., On constant mean curvature hypersurfaces with finite index, Arch. ath. 86 (006), 365 374. R37 (006k:53099) [CY] Cheng, S.Y., Yau, S.T., Hypersurfaces with constant scalar curvature, ath. Ann. 5 (977), 95 04. R043043 (55:4045) [Ch] Chern, S.S., On the curvatures of a piece of hypersurface in Euclidean space, Abh. ath. Seminar der Univ. Hamburg 9 (965), 77 9. R088949 (3:6376) [dcz] do Carmo,.P., Zhou, D., Eigenvalue estimate on noncompact Riemannian manifolds and applications, Transactions Amer. ath. Soc. 35 (999), 39 40. R45597 (000c:53040) [El] Elbert,.F., Constant positive -mean curvature hypersurface, Illinois J. of ath. 46 no. (00), 47 67. R936088 (003g:5303) [ENR] Elbert,.F., Nelli, B., Rosenberg, H., Stable constant mean curvature hypersurfaces, Proc. Amer. ath. Soc. 35, no. 0 (007), 3359 3366. R3768 (008e:5307) [Li] Li, H., Hypersurfaces with constant scalar curvature in space forms, ath. Ann. 305 (996), 665-67. R39970 (97i:53073) [SSY] Schoen, R., Simon, L., Yau, S.T., Curvature estimates for stable minimal hypersurfaces, Acta ath. 34 (975), 75 88. R04363 (54:43) [SZ] Shen, Y. B., Zhu, X. H., On stable complete minimal hypersurfaces in R n+,amer.j. ath. 0 (998), 03 6. R60068 (99c:58040) [Re] Reilly, R.C., Variational properties of functions of the mean curvatures for hypersurfaces in space forms, J. Diff. Geom. 8 (973), 465-477. R03435 (49:60) [Ro] Rosenberg, H., Hypersurfaces of constant curvature in space forms, Bull. Sci. ath. 7, no. (993), 39. R6008 (94b:53097) Instituto de atemática, Universidade Federal de Alagoas, 5707-900 aceió-al, Brazil E-mail address: hilario@mat.ufal.br Instituto de atemática, Universidade Federal do Rio de Janeiro, Caixa Postal 68530, 94-909, Rio de Janeiro-RJ, Brazil E-mail address: walcy@im.ufrj.br Instituto de atemática, Universidade Federal Fluminense, 400-40, Niterói-RJ, Brazil E-mail address: zhou@impa.br