THE BREZIS-NIRENBERG PROBLEM FOR NONLOCAL SYSTEMS

Transcription

1 THE BREZIS-NIRENBERG PROBLEM FOONLOCAL SYSTEMS L.F.O. FARIA, O.H. MIYAGAKI, F.R. PEREIRA, M. SQUASSINA, AND C. ZHANG Abstract. By means of variational methods we investigate existence, non-existence as well as regularity of wea solutions for a system of nonlocal equations involving the fractional laplacian operator and with nonlinearity reaching the critical growth and interacting, in a suitable sense, with the spectrum of the operator. Contents 1. Introduction and results 1. Preliminary stuff 4.1. Notations and setting 4.. Wea solutions 5.3. A priori bounds 5 3. Pohǒzaev identity and nonexistence Proof of nonexistence 9 4. Existence I, subcritical case Proof of existence I Existence II, critical case Preliminary results Proof of existence II 17 References 1 1. Introduction and results Let be a smooth bounded domain. In 1983, Brézis and Nirenberg, in the seminal paper [3], showed that the critical growth semi-linear problem u = λu + u 1 in, 1.1 u > 0 in, u = 0 on, admits a solution provided that λ 0, λ 1 and N 4, λ 1 being the first eigenvalue of with homogeneous Dirichlet boundary conditions and = N/N the critical Sobolev exponent. Furthermore, in dimension N = 3, the same existence result holds provided that µ < λ < λ 1, for a suitable µ > 0 if is a ball, then µ = λ 1 /4 is sharp. By Pohožaev identity, if λ 0, λ 1 and is a star-shaped domain, then problem 1.1 admits no solution. 000 Mathematics Subject Classification. 47G0, 35J50, 35B65. Key words and phrases. Fractional systems, Brezis-Nirenberg problem, nonexistence, regularity. L.F.O. Faria was supported by Fapemig CEX APQ , O.H. Miyagai by CNPq/Brazil and CAPES/Brazil Proc 531/14-3 and F.R. Pereira by Fapemig/Brazil CEX APQ 0097/13 and by the program CAPES/Brazil Proc / Moreover, M. Squassina was partially supported by PROPG/UFJF-Edital 0/014. This paper was partly developed while the second author was visiting the Department of Mathematics of Rutgers University and the fourth author was visiting the University of Juiz de Fora. The hosting institutions are gratefully acnowledged. 1

2 L.F.O. FARIA, O.H. MIYAGAKI, F.R. PEREIRA, M. SQUASSINA, AND C. ZHANG Later on, in 1984, Cerami, Fortunato and Struwe obtained in [6] multiplicity results for the nontrivial solutions of { u = λu + u 1 in, 1. u = 0 on, when λ belongs to a left neighborhood of an eigenvalue of. In 1985, Capozzi, Fortunato and Palmieri proved in [5] the existence of a nontrivial solution of 1. for all λ > 0 and N 5 or for N 4 and λ different from an eigenvalue of. Let s 0, 1 and N > s. The aim of this paper is to obtain a Brézis-Nirenberg type result for the following fractional system s u = au + bv + p p + q u p u v q in, 1.3 s v = bu + cv + q p + q u p v q v in, u = v = 0 in \, where s is defined, on smooth functions, by s ux := CN, s lim ε 0 \B εx ux uy x y N+s dy, x RN, CN, s being a suitable positive constant and p, q > 1 are such that p + q is compared to s := N/N s, the fractional critical Sobolev exponent [8]. The corresponding system in the local case was studied in [1]. For positive solutions, system 1.3 turns into s u = au + bv + p p + q up 1 v q in, 1.4 s v = bu + cv + q p + q up v q 1 in, u > 0, v > 0 in, u = v = 0 in \. In the following we shall assume that is a smooth bounded domain of with N > s and we shall denote by λ i,s the sequence of eigenvalues of s with homogeneous Dirichlet type boundary condition and by µ 1 and µ the real eigenvalues of the matrix a b A :=, a, b, c R. b c Without loss of generality, we will assume µ 1 µ. By solution we shall always mean wea solution in the sense specified in Section, where the functional space X is fully described. It is nown that the first eigenvalue λ 1,s is positive, simple and characterized by 1.5 λ 1,s = inf u X\{0} The following are the main results of the paper. s u dx. u dx

3 THE BREZIS-NIRENBERG PROBLEM FOONLOCAL SYSTEMS 3 Theorem 1.1 Existence I. Assume that b 0, µ < λ 1,s and Then 1.4 admits a solution. p + q < s. Theorem 1. Existence II. Assume that b 0 and Then the following facts hold. p + q = s. 1 If N 4s and 0 < µ 1 µ < λ 1,s then 1.4 admits a solution; If s < N < 4s, there is µ > 0 such that 1.4 admits a solution if µ < µ 1 µ < λ 1,s. Theorem 1.3 Nonexistence. Assume that p + q = s and one of the following facts hold. 1 is star-shaped with respect to the origin and µ < 0; is star-shaped with respect to the origin and A is the zero matrix; 3 b 0 and µ 1 λ 1,s a c. Then 1.4 does not admit any solution. Furthermore, if µ 0 and p + q > s, then 1.4 does not admit any bounded solution if is star-shaped with respect to the origin. Theorem 1.4 Regularity. Assume that p + q s. If u, v is a solution to 1.3, then u, v C 1,α loc for s 0, 1/ and u, v C,α loc for s 1/, 1. In particular u, v solves 1.3 in classical sense. The nonexistence result stated in 3 of Theorem 1.3 holds in any bounded domain. For b = 0 it reads as µ λ 1,s, properly complementing the assertions of Theorem 1.. The above results provide a full extension of the classical results of Brézis and Nirenberg [3] for the local case s = 1. We point out that we adopt in the paper the integral definition of the fractional laplacian in a bounded domain and we do not exploit any localization procedure based upon the Caffarelli-Silvestre extension [4], as done e.g. in []. See [14] for a nice comparison between these two different notions of fractional laplacian in bounded domains. By choosing p = q = s/, system 1.4 reduces to s u = au + bv + u s/n s v N/N s in, s v = bu + cv + u N/N s v s/n s in, 1.6 u > 0, v > 0 in, u = v = 0 in \, which, in the particular case of a = c, setting u = v, boils down to the scalar equation s u = λu + u s 1 in, 1.7 u > 0 in, u = 0 in \, which is the natural fractional counterpart for the classical Brézis-Nirenberg problem [3]. For existence results for this problem, we refer to [1, 13] and to the references therein.

4 4 L.F.O. FARIA, O.H. MIYAGAKI, F.R. PEREIRA, M. SQUASSINA, AND C. ZHANG. Preliminary stuff.1. Notations and setting. We refer the reader to [8] for further details about the functional framewor that follows. For any measurable function u : R we define the Gagliardo seminorm by setting CN, s RN ux uy 1/ 1/. [u] s := x y N+s dxdy = s u dx The second equality follows by [8, Proposition 3.6] when the above integrals are finite. Then, we introduce the fractional Sobolev space H s = {u L : [u] s < }, u H s = u L + [u] s 1/, which is a Hilbert space and we consider the closed subspace.1 X := {u H s : u = 0 a.e. in \ }. Due to the fractional Sobolev inequality, X is a Hilbert space with inner product CN, s ux uyvx vy. u, v X := x y N+s dx dy, R N which induces the norm X = [ ] s. Now, we consider the Hilbert space given by the product.3 Y := X X, equipped with the inner product.4 u, v, ϕ, ψ Y := u, ϕ X + v, ψ X and the norm.5 u, v Y := u X + v X 1/. We shall consider L m L m m > 1 equipped with the standard product norm.6 u, v L m L m := u L m + v L m1/. We recall that we have.7 µ 1 U AU, U R µ U, for all U := u, v R. In this paper, we consider the following notation for product space F F := F and set w + x := max{wx, 0}, w x := min{wx, 0}, for positive and negative part of a function w. Consequently we get w = w + + w. During chains of inequalities, universal constants will be denoted by the same letter C even if their numerical value may change from line to line.

5 THE BREZIS-NIRENBERG PROBLEM FOONLOCAL SYSTEMS 5.. Wea solutions. Consider the following system s u = fu, v in,.8 s v = gu, v in, u = v = 0 in \. where f, g : R R R are Carathéodory mappings which satisfies, respectively, the following growths conditions.9.10 fz, w C1 + z s 1 + w s 1, for all z, w R, gz, w C1 + z s 1 + w s 1, for all z, w R. Definition.1. We say that u, v Y is a wea solutions of.8, if.11 u, v, ϕ, ψ Y = fu, vϕdx + gu, vψdx, for all ϕ, ψ Y..3. A priori bounds. We introduce some notation: for all t R and > 0, we set.1 t := sgnt min{ t, }. From [9, Lemma 3.1] we recall the following Lemma.. For all a, b R, r, and > 0 we have a ba a r b b r 4r 1 r a a r 1 b b r 1. In the following, we prove an L -bound on the wea solutions of.8 which will be needed in order to get nonexistence and regularity results. Lemma.3. Assume that f and g satisfy and let u, v Y be a wea solution to.8. Then we have u, v L. Proof. For all r and > 0, the map t t t r is Lipschitz in R. Then u u r, 0 Y, 0, v v r Y. We test equation.11 with u u r, 0, we apply the fractional Sobolev inequality, Young s inequality, Lemma., and use.9 to end up with.13 u u r 1 L C u u r 1 s X Cr u, u u r r 1 X Cr fu, v u u r dx Cr u u r + u s u r + v s 1 u u r dx Cr u r 1 + u s+r + v s+r dx, for some C > 0 independent of r and > 0. Then, Fatou Lemma, as yields.14 u r Cr L γ r u r 1 + u s +r + v +r s dx

6 6 L.F.O. FARIA, O.H. MIYAGAKI, F.R. PEREIRA, M. SQUASSINA, AND C. ZHANG where γ = s/ 1/ the right hand side may at this stage be. Now, in a similar way, test.11 with 0, v v r to obtain for some C > 0 independent of r.15 v r Cr L γ r v r 1 + u s +r + v +r s dx, the right hand side may be. By.14 and.15 we get.16 u r + L γ r v r Cr L γ r u r 1 + v r 1 + u s+r + v s+r dx. Our aim is to develop a suitable bootstrap argument to prove that u, v L p for all p 1. We start from.16, with r = s + 1 >, and fix σ > 0 such that Crσ < 1. Then there exists K 0 > 0 depending on u and v such that 1 1 s s.17 u s dx + v s dx σ. { u >K 0 } { v >K 0 } By Hölder inequality and.17 we have u s+r dx K s+r 0 { u K 0 } +.18 K s +r 0 + and.19 K s+r 0 + σ u r L γ r By.16,.18 and.19, we have 1.0 u r + L γ r v r L γ r Cr { u >K 0 } u r s dx s v s+r dx K s+r 0 + σ v r L γ r. u s+r dx { u >K 0 } u r 1 + v r 1 dx + K s +r 0. Since r = s + 1, we get u, v L s s +1. We define a sequence {r n } with Since we get r 0 = s + 1, r n+1 = γ r n s +. s + r 0 < s s + 1, u L s +r 0 + v L s +r 0 < +. 1 u s dx s Hence, we aim to begin an iteration in order to get the L -bounds of u and v. Using formula.16 and Hölder inequality, we obtain u L γ r + v L γ r Cr 1 r Cr 1 r s 1 s +r u r s 1 s L s +r + v r 1 L s + u s+r + s+r 1 r +r L s +r v L s +r r 1 u L s + v +r L s + +r u L s + v s +r 1 +r L r s. +r

7 THE BREZIS-NIRENBERG PROBLEM FOONLOCAL SYSTEMS 7 Substituting r n+1 for r, since γ r n = s + r n+1, we get.1 u L γ r n+1 + v L γ r n+1 Cr n+1 1 r n+1 C u L γ + v rn+1 1 rn L γ rn + u L γ + v γ 1 r n r n+1 rn L γ rn Denote Then.1 can be written as T n := max{1, u L γ rn + v L γ rn }.. T n C 1 r n+1 rn+1 1 r n+1 Tn γ rn r n+1. Since r n+1 = γ r n s +, by induction it is possible to prove that r n+1 γ n+ = s 1 + γ n, n N. If n = 0 the assertion follows by a direct calculation. Assume now that the assertion holds for a given n 1 and let s prove it for n + 1. We get which proves the claim. In particular, r n+ γ n+4 = r n+1 γ n+ s γ n+4 = s 1 + γ n s γ n+4 = s 1 + γ n 4, r n+1 γ n+ T n C 1 r n+1 rn+1 1 r n+1 Tn γ rn r n+1 s 1. From., we also have 1 + C 1 1 r n+1 rn+1 r n C 1 1 γ r n 1 γ rn r n+1 rn r n rn T n 1 rn = 1 + C 1+γ 1 γ γ 4 r n 1 r n+1 rn+1 r n+1 rn r n+1 Tn 1 r n C 1+γ +γ 4 + +γ n 1 γ γ 4 γ n r n+1 r r n+1 n+1 rn r n+1 rn 1 r n+1 r1 r n+1 γ n+ 1 n 1 γ = 1 + C 1r n+1 r γn i γ n+ r 0 r n+1 r i+1 T 0 n+1. We can easily compute that γ n+ 1 γ 1r n+1 i=0 Moreover, r i+1 < r 0 γ i+ for every i N, since s 1 s, γ n+ r 0 r n+1 s + 1 s 1. r i+1 γ i+ = r i γ i s γ i+ < r i γ i < < r 0, γ n+ r0 r T 0 n+1

8 8 L.F.O. FARIA, O.H. MIYAGAKI, F.R. PEREIRA, M. SQUASSINA, AND C. ZHANG and r n+1 > γ n+ eventually for n large since r n+1 γ n+ n i=0 1 n r γn i r n+1 i+1 < r 0 γ i+ γn i 1 i=0 γ n+ s 1 > 1, so that r 0 γ i i+ γ i=0 γi=0 i+ < +. Hence T n remains uniformly bounded and the assertion follows. Notice that the L -bound depends on T 0 which depends on u and not only on u s through the presence of K 0 > 0 in estimate Pohǒzaev identity and nonexistence The purpose of this section is to prove Theorem 1.3, for this we need the following auxiliary result nown as Pohǒzaev identity for systems involving the Laplacian fractional operator. Lemma 3.1. Let be a bounded C 1,1 domain and let F C 1 R + R + be such that F u and F v satisfy the growth conditions.9 and.10. Let u, v Y be a solution to system s u = F u u, v in, 3.1 s v = F v u, v in, u = v = 0 in \. Then u, v C s, u, v C 1,α loc for s 0, 1/, u, v C,α loc for s 1/, 1 and u 3. v δ s, δ s C α for some α 0, 1, where δx := distx,, meaning that u/δ s and v/δ s admit a continuous extension to which is C α. Moreover, the following identity holds s u + s v 3.3 dx s F u, vdx Γ1 + [ s u v ] + + N s δ s δ s x, ν dσ = 0, where Γ is the Gamma function. Proof. In light of Lemma.3, we learn that u, v L. Then, F u u, v and F v u, v belong to L too. In turn, by [10, Theorem 1. and Corollary 1.6], we have that u and v satisfy the regularity conclusions stated in 3.. In particular, the system is satisfied in classical sense. Whence, we are allowed to apply [11, Proposition 1.6] to both components u and v, obtaining x u s u dx = s N x v s v dx = s N u s u dx v s v dx Γ1 + s Γ1 + s u δ s x, ν dσ, v δ s x, ν dσ. Then, since s u = F u u, v and s v = F v u, v wealy in and recalling that u s u dx = s u dx, v s v dx = s v dx,

9 THE BREZIS-NIRENBERG PROBLEM FOONLOCAL SYSTEMS 9 we get x uf u u, v dx = s N RN s/ u dx x vf v u, v dx = s N RN s/ v dx Γ1 + s Γ1 + s u δ s x, ν dσ, v δ s x, ν dσ. Observing that F u, v x = F u u, v u x + F v u, v v x, integrating by parts we get, s N s u + s v dx + N F u, vdx [ u v ] = Γ1 + s + δ s δ s x, ν dσ, which concludes the proof Proof of nonexistence. Consider first the case p + q = s with assumption 1 and assume by contradiction that 1.4 admits a positive solution u, v Y. Consider the functions f, g : R + R + R defined by Then, setting fz, w = az + bw + p p + q zp 1 w q, gz, w = bz + cw + q p + q zp w q 1. F z, w = a z + bzw + c w + p + q zp w q = 1 Az, w, z, w R + p + q zp w q, we obtain that F C 1 R + R +, F z = f and F w = g satisfy the growth conditions.9 and.10 and u, v is a wea solution to.8. Then, the components u, v enjoy the regularity 3. stated in Lemma 3.1 and identity 3.3 holds. Testing.11 with ϕ, ψ = u, v, yields s u + s v dx = fu, vu dx + gu, vu dx = AU, U R dx + u p v q dx, which substituted in 3.3, yields, recalling that p + q = s, 1 s Γ1 + [ s u v ] AU, U R dx + + N s δ s δ s x, ν dσ = 0. Since is star-shaped w.r.t. the origin, the equation above yields AU, U Rdx 0. This is a contradiction with.7, because µ < 0 and u, v > 0. Now we cover case. If A is the zero matrix, we get [ u v ] + δ s δ s x, ν dσ = 0, which contradicts the fractional version of Hopf lemma, see [9, Lemma.7], since s u 0 and s v 0 wealy yield u δ s ω and v δ s ω, for some positive constants ω, ω. Let us

10 10 L.F.O. FARIA, O.H. MIYAGAKI, F.R. PEREIRA, M. SQUASSINA, AND C. ZHANG turn to case 3. If ϕ 1 > 0 is the first eigenfunction corresponding to λ 1,s and we assume that a solution of 1.4 exists, by choosing ϕ 1, 0 and 0, ϕ 1 respectively in.11, we get λ 1,s uϕ 1 dx = s s u ϕ1 dx = auϕ1 + bvϕ 1 + p p + q up 1 v q ϕ 1 dx, λ 1,s vϕ 1 dx = s s v ϕ1 dx = buϕ1 + cvϕ 1 + q p + q up v q 1 ϕ 1 dx. Then, since b 0 and u, v > 0, we get λ 1,s uϕ 1 dx > a uϕ 1 dx, λ 1,s vϕ 1 dx > c vϕ 1 dx, that is max{a, c} < λ 1,s. On the other hand, by assumption and a direct calculation λ 1,s a c µ 1 = a + c a c + 4b a + c a c = min{a, c}, which yields max{a, c} λ 1,s, namely a contradiction. Finally we prove the last assertion. In the case p + q > s, any bounded solution of system 1.4 is smooth according to Lemma 3.1 and arguing as above yields the identity 1 s AU, U R dx + 1 s u p v q dx p + q Γ1 + [ s u v ] + + N s δ s δ s x, ν dσ = 0. This yields AU, U R dx > 0, contradicting µ 0 via.7. This concludes the proof. Proof of Theorem 1.4. The assertion follows as a particular case of Lemma Existence I, subcritical case In this section, we will prove the Theorem 1.1 which guarantees the existence of solutions for the problem 1.4 involving subcritical non-linearity Proof of existence I. Let be a bounded domain and suppose that 4.1 b 0, 4. µ < λ 1,s, 4.3 p + q < s. Consider the functional I : Y R defined by IU := 1 U Y 1 AU, U R dx. We shall minimize the functional I restricted to the set { } M := U = u, v Y : u + p v + q dx = 1. By virtue of 4. the embedding X L with the sharp constant λ 1,s, we have 4.4 IU 1 {1, min 1 µ } U Y λ 0. 1,s

11 THE BREZIS-NIRENBERG PROBLEM FOONLOCAL SYSTEMS 11 So define 4.5 I 0 := inf M I, and let U n = u n, v n M be a minimizing sequence for I 0. Then IU n = I 0 +o n 1 C, for some C > 0 where o n 1 0, as n and consequently by 4.4, we get 4.6 [u n ] s + [v n ] s = u n X + v n X = U n Y C. Hence, there are two subsequences of u n X and v n X that we will still label as u n and v n such that U n = u n, v n converges to some U = u, v in Y wealy and [u] s lim inf n [v] s lim inf n CN, s CN, s RN u n x u n y x y N+s dxdy, RN v n x v n y x y N+s dxdy. Furthermore, in view of the compact embedding X L σ for all σ < s cf. [8, Corollary 7.], we get that U n = u n, v n converges to u, v strongly in L p+q, as n. Of course, up to a further subsequence, we have that u n x, v n x converges to ux, vx for a.e. x. Now we will show that U := u, v M. Indeed, since U n M, we have 4.9 u + n p v n + q dx = 1. Since lim n u n p+q dx = u p+q dx, lim n v n p+q dx = v p+q dx, we have in particular u n p+q η 1 and v n p+q η, for some η i L 1 and any n N. Then u + n p xv n + q x p p + q u nx p+q + q p + q v nx p+q η 1 x + η x, for a.e. in. In turn, by the Dominated Convergence Theorem, passing to the limit in 4.9, we obtain u + p v + q dx = 1, and, consequently U = u, v M with u, v 0. We now show that U = u, v is, indeed, a minimizer for I on M and both the components u, v are nonnegative. By passing to the limit in IU n = I 0 + o n 1, where o n 1 0 as n, using 4.7 and 4.8 and the strong convergence of u n, v n to u, v in L, as n, we conclude that IU I 0. Moreover, since U M and I 0 = inf M I IU, we achieve that IU = I 0. This proves the minimality of U M. On the other hand, let GU = u + p v + q dx 1, where U = u, v Y. Note that G C 1 and since U M, G UU = p + q u + p v + q dx = p + q 0,

12 1 L.F.O. FARIA, O.H. MIYAGAKI, F.R. PEREIRA, M. SQUASSINA, AND C. ZHANG hence, by Lagrange Multiplier Theorem, there exists a multiplier µ R such that 4.10 I Uϕ, ψ = µg Uϕ, ψ, ϕ, ψ Y. Taing ϕ, ψ = u, v := U in 4.10, we get U CN, s u + xu y + u xu + y Y = x y N+s dxdy R N CN, s + R N + AU, U R dx. v + xv y + v xv + y x y N+s dxdy Dropping this formula into the expression of IU, we have IU = b v + u + u + v CN, s u + xu y + u xu + y 4.11 dx + 4 R N x y N+s dxdy CN, s v + xv y + v xv + y + 4 x y N+s dxdy 0, R N since b 0, w 0 and w + 0. Furthermore, { IU min 1, 1 µ λ 1 } U Y 0 and using 4.11, we get U = u, v = 0, 0 and therefore u, v 0. We now prove the existence of a positive solution to 1.3. Using again 4.10, we see that U Y AU, U R dx µp + q u p v q dx = 0 and since U M, we conclude that I 0 = IU = µp + q since I 0 is positive, via 4.. Then, by 4.10, U satisfies the following system, wealy, s u = au + bv + pi 0 p + q up 1 v q, s v = bu + cv + qi 0 p + q up v q 1, u = v = 0, \. Now using the homogeneity of system, we get τ > 0 such that W = I 0 τ U is a solution of 1.4. Since b 0 and u, v 0 we get, in wea sense s u au s v cv u 0, v 0 u = v = 0, \. > 0, By the strong maximum principle cf. [9, Theorem.5], we conclude u, v > 0 in.

13 THE BREZIS-NIRENBERG PROBLEM FOONLOCAL SYSTEMS Existence II, critical case Next we turn to Theorem 1., for the critical case p+q = s. The variational tool used is the Mountain Pass Theorem. The embedding X L s is not compact, but we will show that, below a certain level c, the associated functional satisfies the Palais-Smale condition Preliminary results. We will mae use of the following definition 5.1 S s := inf u X\{0} S su, where s u dx R 5. S s u := N ux sdx s is the associated Rayleigh quotient. We also define the following related minimizing problems s u dx R 5.3 S p+q := inf N u X\{0} ux p+q p+q dx and s u + s v dx R 5.4 Sp,q := inf N. u,v X\{0} ux p vx q p+q dx We shall also agree that S s = S p+q, Ss := S p,q, if p + q = s. The following result, in the local case, was proved in [1]. The proof follows by arguing as it was made in [1], but, for the sae of completeness, we present its proof. Lemma 5.1. Let be a domain, not necessarily bounded, and p + q s. Then [ p q p ] p+q p p+q 5.5 Sp,q = + S p+q. q q Moreover, if w 0 realizes S p+q then Bw 0, Cw 0 realizes S p,q, for all positive constants B and C such that B/C = p/q. Proof. Let {w n } X \ {0} be a minimizing sequence for S p+q. Define u n := sw n and v n := tw n, where s, t > 0 will be chosen later on. By definition 5.4, we get s g s wn dx t R 5.6 N S p,q, w n x p+q p+q dx where g : R + R + is defined by gx := x q/p+q + x p/p+q, x > 0.

14 14 L.F.O. FARIA, O.H. MIYAGAKI, F.R. PEREIRA, M. SQUASSINA, AND C. ZHANG The minimum value is assumed by g at the point x = p/q, and it is given by g p q p+q p p p+q p/q = +. q q Whence, by choosing s, t in 5.6 so that s/t = p/q, and passing to the limit, we obtain S p,q g p/qs p+q. In order to prove the reverse inequality, let {u n, v n } X \ {0} be a minimizing sequence for S p,q and define z n := s n v n for some s n > 0 such that u n p+q dx = z n p+q dx. Then, by Young s inequality, we obtain u n p z n q dx p u n p+q dx + q z n p+q dx p + q p + q R N 5.7 = u n p+q dx = z n p+q dx. Thus, using 5.7, we obtain s un + s vn dx u n x p v n x q p+q dx = s q/p+q n s un + s vn dx u n x p z n x q dx p+q s un dx s zn dx s q/p+q n + s p/p+q R n N u n x p+q p+q dx z n x p+q dx gs n S p+q g p/qs p+q. p+q Therefore, letting n in the above inequality, we get the reverse inequality, as desired. From 5.5, the last assertion immediately follows and the proof is concluded. From [7, Theorem 1.1], we learn that S s is attained. Precisely S s = S s ũ, where 5.8 ũx = Equivalently, µ + x x 0 N s S s = inf u X \ {0} u L s = 1, x, R \ {0}, µ > 0, x 0. s u dx = s u dx,

15 THE BREZIS-NIRENBERG PROBLEM FOONLOCAL SYSTEMS 15 where ux = ũx/ ũ L. In what follows, we suppose that, up to a translation, x s 0 = 0 in 5.8. The function x u x := u, x, S 1 s s is a solution to the problem 5.9 s u = u s u in, verifying the property 5.10 u s L s = SN/s s. Define the family of functions U ε x = ε N s u x, x, ε then U ε is a solution of 5.9 and verifies, for all ε > 0, 5.11 s Uε dx = U ε x sdx = S N/s s. Fix δ > 0 such that B 4δ and η C a cut-off function such that 0 η 1 in, η = 1 in B δ and η = 0 in B c δ = RN \ B δ, where B r = B r 0 is the ball centered at origin with radius r > 0. Now define the family of nonnegative truncated functions 5.1 u ε x = ηxu ε x, x, and note that u ε X. The following result was proved in [13] and it constitutes the natural fractional counterpart of those proved for the local case in [3]. Proposition 5.. Let s 0, 1 and N > s. Then the following facts hold. a b c s uε dx S N/s s + Oε N s, as ε 0. C s ε s + Oε N s if N > 4s, u ε x dx C s ε s log ε + Oε s if N = 4s, C s ε N s + Oε s if s < N < 4s, as ε 0. Here C s is a positive constant depending only on s. u ε x sdx = Ss N/s + Oε N, as ε 0. Consider now, for any λ 0, the following minimization problem where S s,λ := inf S s,λv, v X\{0} s v dx λ vx dx R S s,λ v := N R N. vx sdx s

16 16 L.F.O. FARIA, O.H. MIYAGAKI, F.R. PEREIRA, M. SQUASSINA, AND C. ZHANG The following result was proved in [13, Propositions 1 and ] for the first assertion, and in [1, Corollary 8] for the second assertion. Proposition 5.3. Let s 0, 1 and N > s. Then the following facts hold. a For N 4s, S s,λ u ε < S s, for all λ > 0 and any ε > 0 sufficiently small. b For s < N < 4s, there exists λ s > 0 such that S s,λ u ε < S s, for all λ > λ s and any ε > 0 sufficiently small. Proof. For the sae of the completeness, we setch the proof. Case: N > 4s. By Proposition 5., we infer S s,λ u ε SN/s s + Oε N s λc s ε s Ss N/s + Oε N s Case: N = 4s. S s + Oε N s λ C s ε s, S s + ε s Oε N 4s λ C s < S s, for all λ > 0 and ε > 0 small enough and some C s > 0. S s,λ u ε SN/s s + Oε N s λc s ε s log ε + Oε s Ss N/s + Oε N s Case: s < N < 4s. S s + Oε s λ C s ε s log ε, S s + ε s O1 λ C s log ε < S s, for all λ > 0 and ε > 0 small enough and some C s > 0. S s,λ u ε SN/s s + Oε N s λc s ε N s + Oε s Ss N/s + Oε N s S s + ε N s O1 λ C s + Oε s, < S s, for all λ > 0 large enough λ λ s, ε sufficiently small and some C s > 0. This concludes the setch. Even if it is not strictly necessary for the proof of our main result, we state the following Corollary for possible future usage. Corollary 5.4. Suppose that µ 1 given in.7 is positive and let S s,a = inf S s,au, v, u,v X\{0}

17 where S s,a u, v = THE BREZIS-NIRENBERG PROBLEM FOONLOCAL SYSTEMS 17 s u + s v dx Aux, vx, ux, vx R N where p + q = s. Then the following facts holds a If N 4s, then S s,a < S s. ux p vx q dx b For s < N < 4s, there exists µ 1,s > 0, such that if µ 1 > µ 1,s, we have Proof. From Proposition 5.3, we have a For N 4s, we have S s,µ1 u ε < S s, S s,a < S s. s if µ 1 > 0 and provided ε > 0 is sufficiently small. b For s < N < 4s, there exists µ 1,s > 0, such that if µ 1 > µ 1,s, we have S s,µ1 u ε < S s, provided ε > 0 is sufficiently small. R dx Let B, C > 0 be such that B C = p q. From.7 and the above inequalities, we infer that This concludes the proof. S s,a S s,a Bu ε, Cu ε B + C s uε dx µ 1 u ε x dx R N R N / B p C q / s u ε x s sdx R [ N p q/p+q p p/p+q ] = + S s,µ1 u ε q q [ p q/p+q p p/p+q ] < + S s = q q S s. 5.. Proof of existence II. In order to get wea solutions to system 1.4, we now define the functional J : Y R by setting Ju, v = 1 s u + s v dx 1 Au, v, u, v R dx u + p v + q dx, s whose Gateaux derivative is given by 5.13 J u, vϕ, ψ CN, s ux uyϕx ϕy + vx vyψx ψy = R N x y N+s dxdy Au, v, ϕ, ψ R dx p u + p 1 v + q ϕ dx q s v + q 1 u + p ψ dx, s

18 18 L.F.O. FARIA, O.H. MIYAGAKI, F.R. PEREIRA, M. SQUASSINA, AND C. ZHANG for every ϕ, ψ Y. We shall observe that the wea solutions of problem 1.4 correspond to the critical points of the functional J. Under hypothesis 0 < µ 1 µ < λ 1,s, our goal is to prove Theorem 1.. We first show that J satisfies the Mountain Pass Geometry. Proposition 5.5. Suppose µ < λ 1,s. The functional J satisfies a There exist β, ρ > 0 such that Ju, v β if u, v Y = ρ; b there exists e 1, e Y \{0, 0} with e 1, e Y > ρ such that Je 1, e 0. Proof. a By means of.7, using and Poincaré inequality, we have u + p v + q u p+q + v p+q = u s + v s Ju, v 1 1 µ λ 1,s u, v Y C u, v s Y, where C > 0 is a constant. b Choose ũ 0, ṽ 0 Y \ {0, 0} with ũ 0 0, ṽ 0 0 a.e. and ũ 0 ṽ 0 0. Then Jtũ 0, tṽ 0 = t s ũ0 + s ṽ0 dx t RN Aũ 0, ṽ 0, ũ 0, ṽ 0 dx t s ũ p s R 0ṽq 0 dx, N by choosing t > 0 sufficiently large, the assertion follows. This concludes the proof. Therefore, by the previous facts, by the Mountain Pass Theorem it follows that there exists a sequence {u n, v n } Y, so called P S c Palais Smale sequence at level c, such that 5.14 Ju n, v n c, J u n, v n 0, where c is given by with c = inf γ Γ max t [0,1] Jγt, Γ = {γ C[0, 1], Y : γ0 = 0, 0 and Jγ1 0}. Next we turn to the boundedness of {u n, v n } in Y. Lemma 5.6 Boundedness. The P S sequence {u n, v n } Y is bounded. Proof. We have for every n N C + C u n, v n Y Ju n, v n 1 J u n, v n u n, v n s 1 = 1 1 u n, v n Y s 1 Aun, v n, u n, v n dx s R µ u n, v n Y. s λ 1,s Since µ < λ 1,s, the assertion follows. The next result is useful to get nonnegative solutions as wea limits of Palais-Smale sequences. The same argument shows that a critical point of J corresponds to a nonnegative solution to 1.3.

19 THE BREZIS-NIRENBERG PROBLEM FOONLOCAL SYSTEMS 19 Lemma 5.7. Assume that b 0 and µ < λ 1,s. Let {u n, v n } Y be a Palais-Smale sequence for the functional J. Then lim n u n, v n Y = 0. In particular, the wea limit u, v of the PS-sequence {u n, v n } has nonnegative components. Proof. By choosing ϕ := u X and ψ := v X as test functions in 5.13 and using the elementary inequality a ba b a b, for all a, b R, we obtain ux uyu x u y + vx vyv x v y R N x y N+s dxdy RN u x u y + v x v y x y N+s dxdy. Now, note that, since b 0 and w 0 and w + 0, it holds Au, v, u, v R dx Au, v, u, v R dx. In fact, it follows Au, v, u, v R = Au, v, u, v R + bv + u + u + v, Au, v, u, v R. In turn, from the formula for J u, vu, v, it follows that J u, vu, v CN, s RN u x u y + v x v y x y N+s dxdy Au, v, u, v R dx Iu + Iv, where we have set Iw := CN, s RN wx wy x y N+s dxdy µ On the other hand, by the definition of λ 1,s, we have Iw 1 µ [w] λ s, 1,s which finally yields the inequality J u, vu, v 1 µ λ 1,s [u ] s + [v ] s. w dx = [w] s µ w L. Since {u n, v n } Y is a Palais-Smale sequence, we get J u n, v n u n, vn = o n 1, from which that assertion immediately follows. From the boundedness of Palais-Smale sequences see Lemma 5.6 and compact embedding theorems, passing to a subsequence if necessary, there exists u 0, v 0 Y which, by Lemma 5.7, satisfies u 0, v 0 0, such that u n, v n u 0, v 0 wealy in Y as n,

20 0 L.F.O. FARIA, O.H. MIYAGAKI, F.R. PEREIRA, M. SQUASSINA, AND C. ZHANG u n, v n u 0, v 0 a.e. in and strongly in L r for 1 r < s. Recalling that the sequences w 1 n := u + n p 1 v + n q, w n := v + n q 1 u + n p, p + q = s, are uniformly bounded in L s and converges pointwisely to w0 1 = up 1 0 v q 0 and w 0 = v q 1 0 u p 0 respectively, we obtain Hence, passing to the limit in w 1 n, w n w 1 0, w 0, wealy in L s, as n. J u n, v n ϕ, ψ = o n 1, ϕ, ψ Y, as n, we infer that u 0, v 0 is a nonnegative wea solution. sufficient to prove that the solution is nontrivial. Now, to conclude the proof, it is Claim: u 0, v 0 0, 0. Notice that if u 0, v 0 is a solution of system with u 0 = 0, then v 0 = 0. The same holds for the reversed situation. In fact, suppose u 0 = 0. Then, if b > 0, it follows v 0 = 0. If, instead, b = 0, then c {µ 1, µ } < λ 1,s. Since v 0 is a solution of equation s v 0 = cv 0 in, v 0 = 0 on \, we have that v 0 = 0. Therefore, we may suppose that u 0, v 0 = 0, 0. Define, as in [3], L := lim s un + s vn dx, n from J u n, v n u n, v n = o n 1, we get lim n Recalling that Ju n, v n = c + o n 1, thus u + n p v + n q dx = L c = sl N. From the definition of 5.4, we have s un + s vn dx S s and passing to the limit the inequality above, we get L S s L / s. Now, combining this estimate with 5.15, it follows that 5.16 c s N Ss N/s. u n x p v n x q dx Tae B, C > 0 with B/C = p/q and let u ε 0 as in Proposition 5.. Fix ε > 0 sufficiently small that Proposition 5.3 holds and define v ε := u ε / u ε L s. Using the definition of S s,λu, s

21 for every t 0, we obtain THE BREZIS-NIRENBERG PROBLEM FOONLOCAL SYSTEMS 1 JtBv ε, tcv ε t B + C = t B + C S s,µ1 u ε t s B p C q s Thus, an elementary calculation yields s vε dx µ 1 v ε dx t s B p C q ψ max = max ψ = s R + N := ψt, t 0. { B + C N/s B p C q S s,µ1 u ε }. / s By Lemma 5.1 and Proposition 5.3, we conclude that, for ε > 0 small, ψ max < s { B + C } N/s N B p C q S s / s = s { 1 p q/p+q p } p/p+q N/s + ]S s N [ q q = s Ss N/s. N Let now γ C[0, 1], Y be defined by γt := τtbv ε, τtcv ε, t [0, 1], where τ > 0 is sufficiently large that JτBv ε, τcv ε 0. Hence, γ Γ and we conclude that c sup t [0,1] Jγt sup t 0 JtBv ε, tcv ε ψ max < s N s Ss N/s, which contradicts Hence u 0, v 0 0, 0 and the proof is complete. Finally, that u 0 > 0 and v 0 > 0 follows as in the sub-critical case. References [1] C.O. Alves, D.C. de Morais Filho, M.A.S. Souto, On systems of elliptic equations involving subcritical or critical Sobolev exponents, Nonlinear Anal , , 13 [] C. Brändle, E. Colorado, A. de Pablo, U. Sanchez, A concave-convex elliptic problem involving the fractional Laplacian, Proc. Roy. Soc. Edinburgh Sect. A , [3] H. Brézis, L. Nirenberg, Positive solutions of nonlinear elliptic equations involving critical sobolev exponents, Comm. Pure App. Math , , 3, 15, 0 [4] L. Caffarelli, L. Silvestre, An extension problem related to the fractional Laplacian, Comm. Partial Differential Equations 3 007, [5] A. Capozzi, D. Fortunato, G. Palmieri, An existence result for nonlinear elliptic problems involving critical Sobolev exponent, Ann. Inst. H. Poincaré An. Non linéaire 1985, [6] G. Cerami, D. Fortunato, M. Struwe, Bifurcation and multiplicity results for nonlinear elliptic problems involving critical Sobolev exponents, Ann. Inst. H. Poincaré An. Non linéaire , [7] A. Cotsiolis, N.K. Tavoularis, Best constants for Sobolev inequalities for higher order fractional derivatives, J. Math. Anal. Appl , [8] E. Di Nezza, G. Palatucci, E. Valdinoci, Hitchhier s guide to the fractional Sobolev spaces, Bull. Sci. Math , , 4, 11

22 L.F.O. FARIA, O.H. MIYAGAKI, F.R. PEREIRA, M. SQUASSINA, AND C. ZHANG [9] A. Iannizzotto, S. Mosconi, M. Squassina, H s versus C 0 -weighted minimizers, NoDEA Nonlinear Differential Equations Applications 015, , 9, 1 [10] X. Ros-Oton, J. Serra, The Dirichlet problem for the fractional Laplacian: regularity up to the boundary, J. Math. Pures Appl , [11] X. Ros-Oton, J. Serra, The Pohožaev identity for the fractional laplacian, Arch. Ration. Mech. Anal , [1] R. Servadei, E. Valdinoci, A Brezis-Nirenberg result for non-local critical equations in low dimension, Comm. Pure Applied Anal , , 16 [13] R. Servadei, E. Valdinoci, The Brezis-Nirenberg result for the fractional laplacian, Trans. Amer. Math. Soc , , 15, 16 [14] R. Servadei, E. Valdinoci, On the spectrum of two different fractional operators, Proc. Roy. Soc. Edinburgh Sect. A , L.F.O. Faria, O.H. Miyagai, F.R. Pereira Departamento de Matemática Universidade Federal de Juiz de Fora Juiz de Fora, CEP , Minas Gerais, Brazil address: {luiz.faria,olimpio.hiroshi,fabio.pereira}@ufjf.edu.br M. Squassina Dipartimento di Informatica Università degli Studi di Verona Strada Le Grazie 15, I Verona, Italy address: marco.squassina@univr.it C. Zhang Chern Institute of Mathematics Nanai University Tianjin , China address: @mail.nanai.edu.cn