On critical systems involving fractional Laplacian

Transcription

1 J. Math. Anal. Appl Contents lists available at ScienceDirect Journal of Mathematical Analysis and Applications On critical systems involving fractional Laplacian Zhenyu uo a,b,, Senping Luo b, Wenming Zou b a School of Sciences, Liaoning Shihua University, Fushun , China b Department of Mathematical Sciences, Tsinghua University, Beijing , China a r t i c l e i n f o a b s t r a c t Article history: Received 14 June 016 Available online 1 September 016 Submitted by A. Cianchi Keywords: round state Nehari manifold Fractional Sobolev critical exponent Consider the following non-local critical system Δ s u λ 1 u μ 1 u u + αγ u α u v β in, Δ s v λ v μ v v + βγ u α v β v in, u 0, v 0 inr N \, 0.1 where Δ s is fractional Laplacian, 0 < s < 1and all λ 1, λ, μ 1, μ, γ >0, : N N s is a fractional Sobolev critical exponent, N > s, α, β > 1, α + β, and is an open bounded domain in R N with Lipschitz boundary. Under proper conditions, we establish the existence result of the ground state solution to system Elsevier Inc. All rights reserved. 1. Introduction Recently, fractional Sobolev spaces and the corresponding nonlocal equations are applied to many subjects, such as, among others, anomalous diffusion, elliptic problems with measure data, gradient potential theory, minimal surfaces, non-uniformly elliptic problems, optimization, phase transitions, quasigeostrophic flows, singular set of minima of variational functionals, and water waves see [4] and the references therein. For fractional Laplacian, we refer to [1,7 1]. Single equation involving fractional Laplacian had been investigated by some researchers, such as Δ s u λu u u in, u 0 inr N \, which was studied in [1,11]. Supported by NSFC , * Corresponding author. addresses: guozy@163.com Z. uo, luosp1989@163.com S. Luo, wzou@math.tsinghua.edu.cn W. Zou X/ 016 Elsevier Inc. All rights reserved.

2 68 Z. uo et al. / J. Math. Anal. Appl In present paper, we consider the following critical system with fractional Laplacian: Δ s u λ 1 u μ 1 u u + αγ u α u v β in, Δ s v λ v μ v v + βγ u α v β v in, u 0,v0 inr N \, 1.1 where 0 < s < 1and all λ 1, λ, μ 1, μ, γ are positive, : N N s is the fractional Sobolev critical exponent, N>s, α, β >1, α + β ; is an open bounded domain in R N with Lipschitz boundary, and Δ s is fractional Laplacian defined by Δ s ux CN,s R N ux + y+ux y ux y N+s dy, x R N, where CN,s R N 1 cosζ 1 ζ N+s dζ 1 s π N Γ N+s s1 s. 1. Γ s Define the Hilbert space D s as the completion of Cc with respect to the norm Ds induced by the scalar product, Ds given by u, v Ds : CN,s R N ux uy vx vy N+s dxdy. If is an open bounded Lipschitz domain, then D s coincides with the Sobolev space X 0 : f X : f 0a.e.in c }, where X is a linear space of Lebesgue measurable functions from R N to R such that the restriction to of any function f in X belongs to L and the map x, y fx fy N +s is in L R N \ c c, dxdy, and c is the complement of in R N. Usually, there are two ways to define fractional Sobolev space. One is via agliardo seminorm H s R N : u L R N : } ux uy L R N, N +s the other is via Fourier transformation Ĥ s R N : u L R N : R N 1+ ξ s Fuξ dξ<+, and H s R N Ĥs R N. In present paper, we choose the one via agliardo seminorm [u] H s R N : CN,s R N R N ξ s Fuξ dξ. ux uy dxdy N+s

3 Z. uo et al. / J. Math. Anal. Appl The fractional Laplacian operator can be defined by Δ s ux CN,sP.V. R N CN,s lim ε CN,s R N B c ε x F 1 ξ s Fuξ, ux uy dy N+s ux uy dy N+s ux + y+ux y ux y N+s dy where P.V. is the principle value defined by the latter formula. As the norm of fractional Sobolev space H s, we choose not the usual one but the one introduced in [11] u Hs : u + CN,s ux uy N+s dxdy 1, u Hs : u + CN,s R N \ c c ux uy N+s dxdy 1. Consider the fractional Sobolev space equipped with the seminorm D s : x H s R N :u 0 a.e.in c}, u Ds : CN,s R N \ c c ux uy N+s dxdy 1. From u 0a.e. in c, it is easy to see that R N \ c c ux uy dxdy N+s R N ux uy dxdy. N+s Hence, we just denote u Ds by u Ds : CN,s R N ux uy N+s dxdy 1. It can be seen from Lemma. that Ds and Hs are equivalent norms and from Lemma 7 in [9] that D s, Ds is a Hilbert space.

4 684 Z. uo et al. / J. Math. Anal. Appl The solutions of 1.1 coincide with the critical points of the following energy functional E : D s R Eu, v 1 u, v D s 1 λ1 u, + λ v, 1 μ 1 u, + μ v, + γ u α v β, where u p, u p 1 p, 1 p < +, and D s : D s D s is endowed with norm u, v D s : u D s + v D s. Consider the following Nehari manifold M u, v D s \0, 0} : u, v D s λ 1 u, define the ground state energy of 1.1 by + λ v, + μ 1 u, + μ v, + γ B u α v β }, inf Eu, v, 1.3 u,v M and call a solution u, v by a ground state solution if Eu, v B. The existence of ground state solutions to 1.1 is heavily dependent on the following limit system Δ s u μ 1 u u + αγ u α u v β in R N, Δ s v μ v v + βγ u α v β v in R N, 1.4 u, v D s R N. About systems 1.4 and 1.1, our main results are the following. Theorem 1.1. Assume H holds, where H 1 <α,β<, if 4s <N<6s, α, β > 1, if N 6s. Then, 1.4 has a positive ground state solution U, V for all γ >0, which is radially symmetric decreasing with the following decay condition That is, IU, V A, where Ux,Vx C1 + x s N. A inf Iu, v, u,v N Iu, v 1 u, v D s 1 μ1 u + μ v + γ u α v β, N R N u, v D s R N \0, 0} : u, v D s μ1 u + μ v + γ u α v β }. R N

5 Z. uo et al. / J. Math. Anal. Appl Theorem 1.. Assume H holds, is an open bounded domain of R N with Lipschitz boundary, λ 1, λ 0, λs, where λ s is the first eigenvalue of Δ s with homogeneous Dirichlet boundary datum. Then, 1.1 has a positive ground state solution u, v for all γ >0. That is, Eu, v B. As a special case of 1.1, consider the case s 1, α β, N. Then the system 1.1 is reduced to the following Δ 1 u λ 1 u μ 1 u 3 + γ uv in, Δ 1 v λ v μ v 3 + γ u v in, 1.5 u 0,v0 inr \. Note that in this case 4, then the above system is of critical growth. Define S s : inf u D s R N \0} u D s u,r N. Then, by [3], S s is attained by ũx κ ε + x x 0 N s, i.e., Normalizing ũ by ũ,rn, we obtain that Thus, S s S s ū ũ D s ũ,r N. ũ ũ,r N. 1.6 inf u u D s R N D s ū D s u,r N 1 and ū is a positive ground state solution of Δ s u S s u u in R N see Lemma.1. Let U ε x ε N s x u 1, 1.7 ε where u 1 S 1 s ū is a positive ground state solution of Δ s u u u in R N and u 1 D s u 1,R N S N s s. Then, Lemma.1 yields that U ε solves Δ s u u u in R N. Remark 1.3. Here, we obtain u 1 from ū by a multiplier S 1 e.g. Claim 6 in [11]. s ū. Also, we can obtain it by a scaling ū x 1 S s s Consider the limit problem of 1.5: Δ 1 u μ 1 u 3 + γ uv in R, Δ 1 v μ v 3 + γ u v in R. 1.8 Define A 1 : inf I u,v N 1 u, v, B 1 : inf 1 E u,v M 1 1 u, v,

6 686 Z. uo et al. / J. Math. Anal. Appl where N 1 u, v D1 R :u 0, u D 1 R v 0, v D 1 μ 1 u 4 + γ u v, μ v 4 + γ u v }, M 1 u, v D1 : u 0, u D 1 I 1 u,v 1 u, v D E 1 u,v 1 R v 0, v D 1 λ 1 u + μ 1 u 4 + γ u v, λ v + μ v 4 + γ u v }, μ 1 u 4 4,R + μ v 4 4,R + γ u v, R u, v D 1 λ 1 u, λ v, 1 μ 1 u 4 4, + μ v 4 4, + γ u v. 4 Inspired by the arguments of [], we have following results. Theorem If γ <0, then A μ 1 + μ S 1 and A 1 is not attained. If 0 < γ < minμ 1, μ } or γ > maxμ 1, μ }, then A 1 ku ε, lu ε, where U ε is defined by 1.7 and k, l >0 satisfy μ 1 k + γ l 1, γ k + μ l k + ls 1 and A 1 is attained by If minμ 1, μ } γ maxμ 1, μ }, where μ 1 μ, then 1.8 has no nontrivial nonnegative nor nonpositive solution. Theorem 1.5. Assume that 0 < λ 1 λ λ < λ1. 1 If 0 < γ< minμ 1, μ } or γ> maxμ 1, μ }, then B 1 is attained by kw, lw, where k, l >0 satisfy 1.9 and w is a positive ground state of Δ 1 u λu u 3, u D 1. If minμ 1, μ } γ maxμ 1, μ }, where μ 1 μ, then 1.5 has no nontrivial nonnegative nor nonpositive solution. Theorem 1.6. Assume that 0 < λ 1, λ <λ1. 1 B 1 is attained by u, v D1 for any γ <0, where u, v >0. There exists γ 1 0, minμ 1, μ } such that B 1 is attained by u, v D1 for any γ 0, γ 1, where u, v >0.

7 Z. uo et al. / J. Math. Anal. Appl There exists γ maxμ 1, μ } such that B 1 u, v >0. is attained by u, v D1 for any γ>γ, where The paper is organized as following. We give some preliminaries in Section and the proofs of Theorems 1.1 and 1. in Sections 3 and 4, respectively.. Preliminaries We need the following results, which have been proved in [9]. Lemma.1. There exists a positive constant C s such that for any u D s, C s R N where C s is only depending on N and s. ux uy N+s dxdy u,r N u,, Lemma.. For fractional Sobolev space, we have following imbedding results: i H s R N L q R N for any q [, ]; ii D s L q for any q [1, ]; iii D s L q for any q [1,. Remark.3. Noticing the definition of D s, we can rewrite the last two imbedding results as ii D s L q R N for any q [1, ]; iii D s L q R N for any q [1,. Remark.4. By Lemma., it is standard to see that the Nehari Manifold N resp. M is bounded away from 0. Obviously, for any u, v D s R N \0, 0} resp. D s \0, 0}, there exists a unique t u,v such that t u,v u, t u,v v N resp. M. Before going on, we need to establish the following Pohozaev identity see [8] for the scalar equation case. Lemma.5. Assume that u, v D s is a bounded solution of Δ s u u x, u, v in, Δ s v v x, u, v in, u, v D s,.1 where x, 0, 0 0, x, u, v u 0 t x, t, vdt + x, 0,v v 0 t x, u, tdt + x, u, 0,, u, v and x x, u, v are in L 1, u x,, v and v x, u, are locally Lipschitz functions, is a bounded and C 1,1 domain in R N. Then

8 688 Z. uo et al. / J. Math. Anal. Appl N s u, v D s +Γ1+s [ u v ] + δ s δ s x νdσ N x, u, v+ x x x, u, v,. where Γ is the amma function, δx distx,, CN, s is defined in 1. and ν is the unit outward normal vector to at x. Proof. Since x u u x, u, v+x v v x, u, v N x j u j u x, u, v+x j v j v x, u, v j1 N x j u j u x, u, v+v j v x, u, v j1 N j1 x, u, v x j xj x, u, v x j N x j x, u, v j Nx, u, v x x x, u, v. j1 Hence x u u x, u, v+x v v x, u, v x, u, vx ν N x, u, v x x x, u, v..3 Test the first equation of the system by u, we have u u x, u, v Δ s u u CN,s R N R N 1 CN,s u D s. R N R N R N Δ s u u ux uy uxdxdy N+s ux uy dxdy N+s.4 Similarly, we have v v x, u, v v D s..5 On the other hand, by Theorem 1.4 and Proposition 1.6 of [8], we have

9 Z. uo et al. / J. Math. Anal. Appl x u u x, u, v+ x u v x, u, v+ Γ1 + s Γ1 + s u δ s x νdσ s N v δ s x νdσ s N u u x, u, v, v v x, u, v. Therefore, combine with.3,.4 and.5, we get our conclusion. Corollary.6. Let be a bounded set of C 1,1 and star-shaped domain. If N s N u ux, u, v+v v x, u, v x, u, v+ 1 N x xx, u, v.6 for all u, v R and x R N, then.1 admits no positive bounded solution. Moreover, if the inequality in.6 is strict, the system.1 admits no nontrivial bounded solution. For the pure power nonlinearity, the result reads as follows. Corollary.7. Let be a bounded set of C 1,1, and star-shaped domain. Assume that p, q >1, if p +q then the problem Δ s u μ 1 u p+q u + μ 0 p u p uv q in, Δ s v μ v p+q v + μ 0 qu p v q v in, u, v D s, admits no positive bounded solution. Moreover, if p + q> bounded solution. N N s, N N s, then the problem admits no nontrivial Remark.8. Corollaries.6 and.7 are the natural generalization of Corollary 1. and 1.3 in [8]. Lemma.9. Assume that u, v D s is a bounded solution of Δ s u v x, u, v in, Δ s v u x, u, v in, u, v D s,.7 where x, 0, 0 0, x, u, v u t x, t, vdt + x, 0,v t x, u, tdt + x, u, 0, 0 0, u, v and x x, u, v are in L 1 ; u x,, v and v x, u, are locally Lipschitz functions, is a bounded and C 1,1 domain in R N. Then N s u u x, u, v+v v x, u, v+γ1+s u v δ s x νdσ δs v N x, u, v+ x x x, u, v, where Γ is the amma function, δx distx,, and ν is the unit outward normal vector to at x.

10 690 Z. uo et al. / J. Math. Anal. Appl Proof. Since Δ s u + v u x, u, v+ v x, u, v in, Δ s u v v x, u, v u x, u, v in, u, v D s, by Theorem 1.4 and Proposition 1.6 of [8], we have Δ s u + vx u + v s N Δ s u vx u v s N u + v Δ s u + vdx Γ1 + s u + v δ s dσ, u v Δ s u vdx Γ1 + s u v δ s dσ. Submitting the above two identities, we have Δ s ux v+ Δ s vx u s N u u x, u, v+v v x, u, v Γ1 + s u v δ s x νdσ. δs Then the rest of the proof is similar to Lemma.5, we omit the details here. Corollary.10. Let be a bounded, of C 1,1, and star-shaped domain. If N s N u ux, u, v+v v x, u, v x, u, v+ 1 N x xx, u, v,.8 for u, v, R N and x R N, then the system.7 admits no positive bounded solution. For the pure power nonlinearity, the result reads as follows. Corollary.11. Let be a bounded, of C 1,1, and star-shaped domain. Assume p, q >1, if p + q then the problem admits no positive bounded solution. Δ s u μ 1 v p+q 1 + μ 0 qu p v q 1, x, Δ s v μ u p+q 1 + μ 0 pu p 1 v q, x, u v 0, x R N \, N N s,

11 Z. uo et al. / J. Math. Anal. Appl Lemma.1. Let u μ : where ū is defined by 1.6. Then u μ is a positive ground state solution of and the ground state energy is 1 Ss ū,.9 μ Δ s u μ u u in R N.10 A μ : 1 u μ D s 1 μ u μ s,r N N S s μ..11 Proof. It is standard to see that the infimum S s is attainable if and only if the equation.10 has a ground state solution. So, we only need to verify that u μ is a solution of.10. By standard minimization theory, there exists a Lagrange multiplier l such that CN,s R N ūx ūy φx φy N+s dxdy l for any φ D s R N. Testing by φ ū, we obtain that Then, for any φ D s R N, we have CN,s R N By.1, for any φ D s R N, we deduce that CN,s R N Ss μ l l ū,r N ū Ds S s. ūx ūy φx φy N+s dxdy S s uμ x u μ y φx φy 1 N+s dxdy CN,s R N R N ū ūφ ūx ūy φx φy 1 Ss Ss ū ūφ μ R N 1 1 Ss μ Ss u μ u μ φ μ S s μ R N u μ u μ φ, R N R N ū ūφ..1 N+s dxdy which means that u μ is a solution of.10. Finally,.11 follows from a direct computation. This completes the proof.

12 69 Z. uo et al. / J. Math. Anal. Appl Proof of Theorem 1.1 Consider Δ s u μ 1 u u + αγ u α u v β, x B R, Δ s v μ v v + βγ u α v β v, x B R, u, v D s B R, 3.1 where γ >0and B R : x R N : x < R}. Define N R : u, v D s B R \0, 0} : u, v D s } μ 1 u,b R + μ v,b R + γ u α v β 3. B R and AR : inf u,v N R Iu, v. 3.3 For ε [0, minα, β} 1, consider Δ s u μ 1 u ε u + α εγ ε u α ε u v β ε, x, Δ s v μ v ε v + β εγ ε u α ε v β ε v, x, u, v D s. 3.4 Define I ε u, v : 1 u, v D s 1 μ1 u ε + μ v ε + γ u α ε v β ε, ε N ε : u, v D s \0, 0} : ε u, v : u, v Ds μ1 u ε + μ v ε + γ u α ε v β ε 0}, 3.5 and A ε : Lemma 3.1. For ε 0, minα, β} 1, there holds inf u,v N ε I ε u, v. 3.6 } A ε < min inf I ε u, 0, inf I ε 0,v. u,0 N ε 0,v N ε Proof. Since minα, β}, we see that < ε <. Then, we may assume that u i is a ground state solution of

13 Z. uo et al. / J. Math. Anal. Appl Δ s u μ i u ε u, u D s, i 1,. Thus, I ε u 1, 0 a 1 : inf I ε u, 0, I ε 0,u a : inf I ε 0,v. u,0 N ε 0,v N ε It is claimed that, for any τ R, there exists a unique tτ > 0such that In fact, tτu1, tττ u N ε. tτ ε u 1 D s + τ u D s μ 1 u 1 ε + τ ε μ u ε + τ β ε γ u 1 α ε u β ε pa 1 + pa τ pa 1 + pa τ ε + τ β ε γ u 1 α ε u β ε, where p ε ε. Noting that t0 1 and, we deduce that i.e., lim τ 0 t τ τ β ε τ ε B β 1 γ u 1 α ε u β ε, εa 1 t τ β ε γ u 1 α ε u β ε εa 1 τ β ε τ 1+o1, as τ 0. Therefore, and then, tτ 1 tτ ε 1 γ u 1 α ε u β ε εa 1 τ β ε 1+o1, as τ 0, γ u 1 α ε u β ε a 1 τ β ε 1+o1, as τ 0. Noticing 1 1 p 1 ε, we get that A ε I ε tτu1, tττ u 1 1 pa 1 + pa τ ε + τ β ε γ u 1 α ε u t β ε ε ε 1 a 1 1 p τ β ε γ u 1 α ε u β ε + o τ β ε <a 1 inf u,0 N ε I ε u, 0 for τ small enough. Similarly, A ε < inf 0,v Nε I ε 0, v, which completes the proof.

14 694 Z. uo et al. / J. Math. Anal. Appl Noticing the definitions of u μ1 and u μ in.9, similarly as Lemma 3.1, we have } A<min inf Iu, 0, inf I0,v u,0 N 0,v N min Iu μ1, 0,I0,u μ } s N s s min μ 1 S N s s, s N s } s μ S N s s. N N 3.7 Lemma 3.. For any ε 0, minα, β} 1, system 3.4 has a positive ground state solution u ε, v ε, and u ε, v ε are radially symmetric decreasing. Proof. It is standard to see that A ε > 0. For u, v N ε with u 0, v 0, we denote by u, v as its symmetric radial decreasing rearrangement. By Theorem 1.1 in [6] and the properties of symmetric radial decreasing rearrangement, we obtain that u,v D s u, v D s μ1 u ε + μ v ε + γ u α ε v β ε μ1 u ε + μ v ε + γ u α ε v β ε. Hence, there exists t 0, 1] such that t u, t v N ε. Then, we have I ε t u, t v 1 1 t u,v D ε s ε ε u, v D s I ε u, v. 3.8 Thus, we may take a minimizing sequence u n, v n N ε of A ε such that u n, v n u n, vn and I ε u n,v n A ε as n. Evidently, 3.8 guarantees that u n, v n are uniformly bounded in D s. Passing to a subsequence, we may assume that u n u ε, v n v ε weakly in D s. Since D s L ε, we deduce that μ1 u ε ε + μ v ε ε + γ u ε α ε v ε β ε lim n μ1 u n ε + μ v n ε + γ u n α ε v n β ε ε ε lim n I εu n,v n ε ε A ε > 0, which means that u ε, v ε 0, 0. Moreover, u ε 0, v ε 0are radially symmetric decreasing. Noticing u ε, v ε D s lim n u n, v n D s, we have u ε,v ε D s μ1 u ε ε + μ v ε ε + γ u ε α ε v ε β ε.

15 Z. uo et al. / J. Math. Anal. Appl So, there exists t ε 0, 1] such that t ε u ε, t ε v ε Nε, and then A ε I ε tε u ε, t ε v ε 1 1 t ε u ε,v ε D ε s ε ε lim n u n,v n D s lim n I εu n,v n A ε, which implies that t ε 1, u ε, v ε N ε, Iu ε, v ε A ε, and u ε,v ε D s lim n u n,v n D s. That is, u n u ε, v n v ε strongly in D s. It follows from the standard minimization theory that there exists a Lagrange multiplier L R such that I εu ε,v ε +L εu ε,v ε 0, where ε is given in 3.5. Since I εu ε, v ε u ε, v ε ε u ε, v ε 0and εu ε,v ε u ε,v ε ε μ1 u ε ε + μ v ε ε + γ u ε α ε v ε β ε < 0, we get that L 0and hence I εu ε, v ε 0. Note that A ε Iu ε, v ε and Lemma 3.1 yield that u ε 0 and v ε 0. Since u ε, v ε 0are radially symmetric decreasing, by the regularity theory [7,1] and Proposition.17 in [1], we have u ε, v ε > 0in. This completes the proof. Proof of Theorem 1.1. We claim that AR A for all R> Indeed, assume R 1 <R. It follows from N R 1 NR that AR AR 1. On the other hand, for every u, v NR, define u1 x,v 1 x R N s R R : u x, R 1 R 1 and obviously, u 1, v 1 NR 1. Then, we have N s R 1 AR 1 Iu 1,v 1 Iu, v, u, v NR, R v x, R 1 which implies that AR 1 AR. Hence, AR 1 AR. Let u n, v n N be a minimizing sequence of A. We may assume that u n, v n D s BRn for some Rn > 0. Then u n, v n NR n and A lim n Iu n,v n lim n AR nar. This, combining A AR, completes the proof of the claim.

16 696 Z. uo et al. / J. Math. Anal. Appl For every u, v N1, by 3. and 3.5, there exists t ε > 0with t ε 1as ε 0such that tε u, t ε v N ε. Therefore, lim sup ε 0 A ε lim sup I ε tε u, t ε v Iu, v, ε 0 u, v N1. Then, 3.9 ensures that lim sup A ε A1 A ε 0 Since Lemma 3., we may assume u ε, v ε is a positive ground state solution of 3.4, which is radially symmetric decreasing. It follows from 3.5 and Lemma. that A ε ε ε u ε,v ε D s C>0, ε 0, minα, β} 1], 3.11 where C is independent of ε. Then, it follows from 3.10 that u ε, v ε are uniformly bounded in D s. Passing to a subsequence, we may assume that u ε u 0, v ε v 0 weakly in D s. Then u 0, v 0 is a solution of Δ s u μ 1 u u + αγ u α u v β, x, Δ s v μ v v + βγ u α v β v, x, u, v D s. 3.1 Suppose by contradiction that u ε + v ε is uniformly bounded. Then, the Dominated Convergent Theorem guarantees that lim ε 0 uε ε lim ε 0 u 0, lim ε 0 u α ε ε vε β ε vε ε u α 0 v β 0. Combining these with I εu ε, v ε I u 0, v 0, similarly as the proof of Lemma 3., we have that u ε u 0, v ε v 0 strongly in D s ensures that u 0, v 0 0, 0, and moreover, u 0 0, v 0 0. Without loss of generality, we may assume that u 0 0. Then Proposition.17 in [1] yields that u 0 > 0in. By Pohozaev identity., we derive a contradiction 0 < Γ1 + s N s [ u v ] + δ s δ s x νdσ v 0, μ1 u + μ v + γ u α v β u, v D s 0, where δx distx, and ν is the outward unit normal vector on. Therefore, u ε + v ε, as ε 0. Define M ε : maxu ε 0, v ε 0}. Noticing u ε 0 max B1 u ε x and v ε 0 max B1 v ε x, we get that M ε +, as ε 0. Define

17 Z. uo et al. / J. Math. Anal. Appl Then, we see that and U ε, V ε is a solution of From U ε x :Mε 1 u ε M q ε ε x, V ε x :Mε 1 v ε M q ε ε x, q ε : ε. s maxu ε 0,V ε 0} max Δ s U ε μ 1 U ε 1 ε Δ s V ε μ V ε 1 ε U ε,v ε D s B M qε. ε U ε D s max x B M qε ε U ε x, + α εγ ε U ε α 1 ε + β εγ ε U ε α ε M sq ε+nq ε ε u ε D s max x B M qε ε } V ε x V β ε ε, x B M qε ε, V β 1 ε ε, x B M qε ε, M N sε s ε u ε D s u ε D s, V ε D s v ε D s it follows that U ε, V ε is bounded in D s B M qε. Theorem 1.4 in [5] ensures that U ε ε, V ε is Hölder continuous. By Arzela Ascoli Theorem, we see that, up to a subsequence, U ε, V ε U, V D s R N uniformly in every compact subset of R N as ε 0, and U, V is a solution of 1.4, i.e., I U, V 0. Furthermore, U 0, V 0are radially symmetric decreasing. By 3.13, we see that U, V 0, 0 and then U, V N. Then, by 3.10, we have A IU, V 1 1 U, V D s 1 lim inf ε 0 1 U ε,v ε D s lim inf ε 0 lim inf A ε ε 0 A, 1 1 u ε,v ε D ε s which implies that IU, V A. 3.7 ensures that U 0and V 0. The regularity theory [7,1] and Proposition.17 in [1] guarantees that U > 0and V > 0, that is, U, V is a positive ground state solution of 1.4, which is radially symmetric decreasing. By Theorem 4 in [13], we see that Ux, V x C1 + x s N. Remark 3.3. It is claimed that if γ<0, then 1.4 has no nontrivial ground state solution, which is the reason that we only consider the case γ >0. In fact, if γ <0, then we have which implies that u D s μ 1 u,r N αγ R N u α v β 0, μ 1 u,r N u D s S s u,r N.

18 698 Z. uo et al. / J. Math. Anal. Appl If u D s R N \0}, then u,r N we get that 1 S s μ 1, which yields that u D s S Ss s μ 1. Therefore, by.11, 1 A μ1 1 S s μ 1 s N u D s. Similarly, A μ s N v D s for any v D s R N \ 0}. Suppose that u, v D s R N \ 0, 0} is a ground state solution of 1.4. Then, A Iu, v s u N Ds + v D s A μ, if u 0,v 0, A μ1 + A μ, if u 0,v 0, A μ1, if u 0,v 0. It can be seen that A mina μ1, A μ }, which means that 1.4 has no nontrivial ground state solution. 4. Proof of Theorem 1. In this section we assume γ >0. Define the following infimum S s : inf u,v D s R N u,v 0,0 u, v D s μ 1 u + μ,r N v + γ u,r N R α v β N, S s,λ1,λ : inf u,v D s u,v 0,0 u, v D s λ 1 u, λ v, μ 1 u, + μ v, + γ u α v β By Theorem 1.1, it can be seen that S s is attained by U, V D s R N, where U and V are radially symmetric decreasing and satisfy Fix δ >0such that B 4δ and define Ux,Vx C1 + x s N.. u ε x :ηxu ε x, v ε x :ηxv ε x, where η is a cut-off function satisfying η Bδ 1, η B c δ 0, and Then, we have Lemma 4.1. U ε x :ε N s x U, V ε x :ε N s x V. ε ε u ε D s U ε D s + Oε N s U D s + Oε N s, 4.1 v ε D s V ε D s + Oε N s V D s + Oε N s, 4. u ε, U ε,r N + Oε N U,R N + Oε N, 4.3

19 Z. uo et al. / J. Math. Anal. Appl where C is a positive constant relevant to s. v ε, V ε + Oε N V,R N + Oε N, 4.4,R N u α ε vε β Uε α Vε β + Oε N U α V β + Oε N, 4.5 R N R N Cε s + Oε N s, if N>4s, u ε,, v ε, Cε s ln ε + Oε s, if N 4s, 4.6 Cε N s + Oε s, if N<4s, Proof. Noting that Ds and,rn are invariant under the scaling, it is easy to see that the signs in hold. It is standard to see that the sign in hold, so the proofs are omitted. For the sign in 4.1 and 4., we refer to Proposition 1 in [11]. For the readers convenience, we give the details here. It is claimed that u ε x u ε y Cε N s, x R N,y B c δ with δ. 4.7 Indeed, for any x R N, y Bδ c with δ, assume that z is any point on the segment joining x and y, i.e., z tx +1 ty for some t [0, 1]. Then, z y + tx y y t δ t δ δ. Since U ε z Cε N s which implies that We claim that 1 + z s N, ε we deduce that u ε z U ε z ηz+ηz U ε z Cε N s 1+ z N s z N s 1 ε ε ε Cε N s 1+ z N s z + 1+ z N s 1 δ ε ε Cε N s 1+ δ Cε N s, ε 1+ z ε N s u ε x u ε y u ε z Cε N s. u ε x u ε y Cε N s min1, }, x, y B c δ. 4.8 In fact, let x, y Bδ c. If δ, then 4.8 follows from 4.7. Hence, we consider the case > δ. Noticing U ε x Cε N s 1 + x s N ε, we have u ε x U ε x Cε N s, x B c δ. 4.9

20 700 Z. uo et al. / J. Math. Anal. Appl Therefore, for x, y Bδ c, we get that u ε x u ε y u ε x+u ε y Cε N s, which means that 4.8 holds. Noticing η B c δ 0, by 4.8, we have B c δ Bc δ Cε N s u ε x u ε y N+s dxdy Cε N s O ε N s. x <δ x y <1 dxdy + N+s B δ B δ x <δ x y >1 min1, } N+s dxdy 1 dxdy N+s 4.10 Letting L x, y R N : x B δ,y Bδ c and < δ }, x, y R N : x B δ,y Bδ c and > δ }, then, by 4.7, we see that L u ε x u ε y N+s dxdy Cε N s x <δ x y < δ dxdy N+s 4.11 O ε N s. It follows from 4.9 that U ε xv ε y Cε N s 1+ x ε s N N s ε C 1+ x ε for any x R N and y Bδ c. For x, y, by 4.9 and 4.1, we deduce that u ε x u ε y N+s dxdy + U ε x U ε y N+s dxdy + + U ε x u ε y N+s dxdy U ε y u ε y N+s dxdy U ε x U ε y U ε y u ε y N+s dxdy U ε x U ε y N+s dxdy + Uε y+u ε y N+s dxdy U ε x U ε y u ε y + U ε y U ε y u ε y N+s dxdy s N 4.1

21 +4 + C Z. uo et al. / J. Math. Anal. Appl U ε x U ε y N+s dxdy +4 Uε y dxdy N+s U ε xu ε y+u ε y N+s dxdy 4.13 U ε x U ε y N+s dxdy + Cε N s x <δ x y > δ 1+ x ε x <δ x y > δ N+s N s dxdy U ε x U ε y N+s dxdy + Cε N s + Cε N 1 + ζ N+s dζ ζ <δ dζ 1 dxdy N+s ξ > δ ξ N s dξ ξ N s dξ ζ < δ ε ξ > δ U ε x U ε y N+s dxdy + Cε N s + Cε N U ε x U ε y N+s dxdy + O ε N s. ζ < δ ε ζ N+s dζ Combining 4.10, 4.11 and 4.13, we obtain that R N u ε x u ε y N+s dxdy B δ B δ + L B δ B δ R N U ε x U ε y N+s dxdy + B c δ Bc δ u ε x u ε y N+s dxdy + U ε x U ε y N+s dxdy + U ε x U ε y N+s dxdy + O ε N s, u ε x u ε y N+s dxdy u ε x u ε y N+s dxdy U ε x U ε y N+s dxdy + O ε N s which proves 4.1. Similarly, 4. holds. Lemma 4.. If 0 < λ 1, λ <λ s, then S s < min μ 1 S s,λ1, μ S s,λ }, where S s,λ : inf u D s \0} u D s λ u, u.,

22 70 Z. uo et al. / J. Math. Anal. Appl Proof. By [1,11], we may assume that w μ1 achieves S s,λ1 with w μ1, t uμ1,ɛu μ1, i.e., 1 Ss,λ1 μ 1. Define tɛ : tɛ wμ1 D s,λ 1 + ɛ w μ1 1 D s,λ μ1 + μ ɛ + γ ɛ β w μ1,, where D s,λ : D s λ,. It is easy to see that tɛw μ1, tɛɛw μ1 M. Since wμ1 D s,λ 1 μ 1 w μ1, and t0 1, we have i.e., lim ɛ 0 t ɛ ɛ β ɛ γβ, μ 1 t ɛ γβ ɛ β ɛ μ 1 1+o1, as ɛ 0. Thus, tɛ 1 γ ɛ β μ 1 1+o1, as ɛ 0, and so, tɛ 1 γ ɛ β μ 1 1+o1, as ɛ 0. Then, we obtain that tɛ μ 1 + μ ɛ + γ ɛ β w μ1, 1 γ ɛ β 1+o1 μ1 + μ ɛ + γ ɛ β w μ1 μ, 1 μ 1 w μ1, γ ɛ β w μ1, + o ɛ β <μ 1 w μ1, for ɛ small enough. Hence, S s Similarly, S s <μ S s,λ. inf u,v M μ 1 u, + μ v, + γ u α v β tɛ μ 1 + μ ɛ + γ ɛ β w μ1, <μ1 w μ1, μ 1 S s,λ1.

23 Z. uo et al. / J. Math. Anal. Appl Proof of Theorem 1.. It follows from Lemma 4.1 that S s,λ1,λ u ε D s λ 1 u ε, + v ε D s λ v ε, μ 1 u ε, + μ v ε, + γ uα ε vε β U D s + V D s Cε + Oε N s μ 1 U + μ,r N V + γ U,R N R α V β + Oε N N < S s Let u n, v n } be a minimizing sequence for S s,λ1,λ normalized by that is, μ 1 u n, + μ v n, + γ u n α v n β 1, u n,v n D s λ 1 u n, λ v n, S s,λ1,λ + o Noting that u n } and v n } are bounded in D s, by Lemma., we may extract two subsequences-still denoted by u n } and v n }-such that u n u, v n v weakly in D s, u n u, v n v strongly in L, u n u, v n v a.e. on, with μ 1 u, + μ v, + γ u α v β 1. Setting w n : u n u and z n : v n v, then w n 0, z n 0weakly in D s and w n 0, z n 0a.e. on. Then, 4.15 yields that S s,λ1,λ + λ 1 u n, + λ v n, + o1 u n,v n D s S s. By 4.14, we see that λ 1 u, + λ v, S s S s,λ1,λ w n 0, z n 0weakly in D s, we have > 0, which means that u, v 0, 0. Since u n D s w n D s + u D s + o1, v n D s z n D s + v D s + o1. Then, 4.15 ensures that S s,λ1,λ w n D s + u D s λ 1 u, + z n D s + v D s λ v, + o The Brezis Lieb Lemma guarantees that

24 704 Z. uo et al. / J. Math. Anal. Appl μ 1 u + w n, + μ v + z n, + γ μ 1 u, + μ v, + γ u α v β + μ 1 w n, + μ z n, + γ u + w n α v + z n β w n α z n β + o1. Since and μ 1 u, + μ v, + γ μ 1 w n, + μ z n, + γ u α v β 1 w n α z n β 1, we see that 1 μ 1 u, + μ v, + γ u α v β + μ 1 w n, + μ z n, + γ w n α z n β μ 1 u, + μ v, + γ u α v β + 1 Ss w n,z n D s + o1. + o It follows from 4.16, 4.17, 4.14 and S s,λ1,λ > 0that u D s λ 1 u, + v D s λ v, S s,λ1,λ μ 1 u, + μ v, + γ u α v β Ss,λ1,λ + 1 w n,z n S D s + o1 s S s,λ1,λ μ 1 u, + μ v, + γ u α v β + o1, which, combining u, v 0, 0, implies that u D s λ 1 u, + v D s λ v, μ 1 u, + μ v, + γ S s,λ1,λ. u α v β

25 Z. uo et al. / J. Math. Anal. Appl Therefore, S s,λ1,λ is attained by u, v. It remains to show that u, v can not be the type of u, 0 or 0, v. Suppose by contradiction that S s,λ1,λ is attained by u, 0. Then, S s,λ1,λ u D s λ 1 u, μ 1 u, μ 1 S s,λ1, which contradicts to Lemma 4.. Hence, u, v can not be the type of u, 0. Similarly, it can not be 0, v. Thus, we have proved that 1.1 has a nontrivial ground state solution u, v via the infimum S s,λ1,λ. Meanwhile, we may prove it via a PS sequence for B or a minimizing sequence for B, where B is given by 1.3 and Here for some e D. It is standard to see that B : inf sup h Γ 0<t<1 E ht. Γ: h C [0, 1], D : h0 0,h1 e e 1,e } B B B, 4.18 where B inf u,v D\0,0} sup t>0 Etu, tv. Since 4.18, what we have done in this proof is equivalent to that for any minimizing sequence u n, v n } of E, we may obtain a weak limit u, v such that B Eu, v, where u 0and v 0. That is, u, v is a nontrivial ground state solution of 1.1. To get a positive ground state solution, we claim that u Ds u Ds. Indeed, R N ux uy dxdy N+s R N R N ux uy N+s dxdy ux uy uxuy N+s dxdy 0. Then, for the minimizing sequence u n, v n M, we have u n + v D n u s D n s D s + v n D s λ1 u n + λ v n + μ 1 u n + μ v n + γ u n α v n β, which yields that there exist t n 0, 1] such that t n u n, t n v n M. Hence, we may choose a minimizing sequence by ū n, v n t n u n, t n v n, and the weak limit ū, v is nonnegative. The maximum principle see Proposition.17 in [1] guarantees that ū >0and v >0, which completes the proof. Remark 4.3. Note that, to get a positive solution, we can not solve it by u, v M u, v M, as what is done for usual Laplacian. To solve it, besides what we have done, we can also do it in the following way. Similar to Lemma 6 in [10], we have if u, v D s satisfies Δ s u 0and Δ s v 0in the weak sense, then u 0, v 0in. Thus, by Proposition.17 in [1], we have u > 0, v >0in.

26 706 Z. uo et al. / J. Math. Anal. Appl Remark 4.4. Even if S s,λ1,λ γ u α v β 1, we get that 0, it is also attained. Indeed, by 4.16 and μ 1 u, + μ v, + u D s λ 1 u, + v D s λ v, S s,λ1,λ S s,λ1,λ μ 1 u, + μ v, + γ u α v β. References [1] B. Barrios, E. Colorado, A. de Pablo, U. Sánchez, On some critical problems for the fractional Laplacian operator, J. Differential Equations [] Z. Chen, W. Zou, Positive least energy solutions and phase separation for coupled Schrödinger equations with critical exponent, Arch. Ration. Mech. Anal [3] A. Cotsiolis, N.K. Tavoularis, Best constants for Sobolev inequalities for higher order fractional derivatives, J. Math. Anal. Appl [4] E. Di Nezza,. Palatucci, E. Valdinoci, Hitchhiker s guide to the fractional Sobolev spaces, Bull. Sci. Math [5] C. Mou, Nonlinear elliptic systems involving the fractional Laplacian in the unit ball and on a half space, Commun. Pure Appl. Anal [6] Y.J. Park, Fractional Polya Szegö inequality, J. Chungcheong Math. Soc [7] X. Ros-Oton, J. Serra, The Dirichlet problem for the fractional Laplacian: regularity up to the boundary, J. Math. Pures Appl [8] X. Ros-Oton, J. Serra, The Pohozaev identity for the fractional Laplacian, Arch. Ration. Mech. Anal [9] R. Servadei, E. Valdinoci, Mountain pass solutions for non-local elliptic operators, J. Math. Anal. Appl [10] R. Servadei, E. Valdinoci, Weak and viscosity solutions of the fractional Laplace equation, Publ. Mat [11] R. Servadei, E. Valdinoci, The Brezis Nirenberg result for the fractional Laplacian, Trans. Amer. Math. Soc [1] L. Silvestre, Regularity of the obstacle problem for a fractional power of the Laplace operator, Comm. Pure Appl. Math [13] X. Yu, Liouville type theorems for integral equations and integral systems, Calc. Var. Partial Differential Equations