FRACTIONAL ELLIPTIC SYSTEMS WITH NONLINEARITIES OF ARBITRARY GROWTH

Transcription

1 Electronic Journal of Differential Equations, Vol (2017, No. 206, ISSN: URL: htt://ejde.math.txstate.edu or htt://ejde.math.unt.edu FRACTIONAL ELLIPTIC SYSTEMS WITH NONLINEARITIES OF ARBITRARY GROWTH EDIR JUNIOR FERREIRA LEITE Communicated by Mokhtar Kirane Abstract. In this article we discuss the existence, uniqueness and regularity of solutions of the following system of couled semilinear Poisson equations on a smooth bounded doma in R n : A s u = v A s v = f(u u = v = 0 on where s (0, 1 and A s denote sectral fractional Lalace oerators. We assume that 1 < < 2s, and the function f is suerlinear and with no n 2s growth restriction (for examle f(r = re r ; thus the system has a nontrivial solution. Another imortant examle is given by f(r = r q. In this case, we rove that such a system admits at least one ositive solution for a certain set of the coule (, q below the critical hyerbola q + 1 = n 2s n whenever n > 2s. For such weak solutions, we rove an L estimate of Brezis-Kato tye and derive the regularity roerty of the weak solutions. 1. Introduction and statement of main result This work is devoted to the study of existence and uniqueness of solutions for nonlocal ellitic systems on bounded domains which will be described henceforth. The sectral fractional Lalace oerator A s is defined in terms of the Dirichlet sectra of the Lalace oerator on. Roughly seaking, if (ϕ k denotes a L 2 - orthonormal basis of eigenfunctions corresonding to eigenvalues (λ k of the Lalace oerator with zero Dirichlet boundary values on, then the oerator A s is defined as A s u = k=1 c kλ s k ϕ k, where c k, k 1, are the coefficients of the exansion u = k=1 c kϕ k. A closely related to (but different from the sectral fractional Lalace oerator A s is the restricted fractional Lalace oerator ( s, see [27, 29]. This is defined as ( s u(x = C(n, s P. V. R n u(x u(y dy, x y n+2s 2010 Mathematics Subject Classification. 35J65, 49K20, 35J40. Key words and hrases. Fractional ellitic systems; critical growth; critical hyerbola. c 2017 Texas State University. Submitted June 10, Published Setember 7,

2 2 E. J. F. LEITE EJDE-2017/206 for all x R n, where P.V. denotes the rincial value of the first integral and ( 1 cos(ζ 1 1 C(n, s = R ζ n+2s dζ n with ζ = (ζ 1,..., ζ n R n. We remark that ( s is a nonlocal oerator on functions comactly suorted in R n, i.e., to check whether the equation holds at a oint, information about the values of the function far from that oint is needed. Fractional Lalace oerators arise naturally in several different areas such as Probability, Finance, Physics, Chemistry and Ecology, see [1, 5]. These oerators have attracted secial attention during the last decade. An extension for sectral fractional oerator was devised by Cabré and Tan [6] and Caella, Dávila, Duaigne, and Sire [7] (see Brändle, Colorado, de Pablo, and Sánchez [4] and Tan [31] also. Thanks to these advances, the boundary fractional roblem A s u = u u = 0 on (1.1 has been widely studied on a smooth bounded oen subset of R n, n 2, s (0, 1 and > 0. Particularly, a riori bounds and existence of ositive solutions for subcritical exonents ( < n+2s n 2s has been roved in [4, 6, 8, 9, 31] and nonexistence results has also been roved in [4, 30, 31] for critical and suercritical exonents ( n+2s n 2s. The regularity result has been roved in [7, 31, 32]. When s = 1/2, Cabré and Tan [6] established the existence of ositive solutions for equations having nonlinearities with the subcritical growth, their regularity, the symmetric roerty, and a riori estimates of the Gidas-Sruck tye by emloying a blow-u argument along with a Liouville tye result for the square root of the Lalace oerator in the half-sace. Then [31] has the analogue to 1/2 < s < 1. Brändle, Colorado, de Pablo, and Sánchez [4] dealt with a subcritical concaveconvex roblem. For f(u = u q with the critical and suercritical exonents q n+2s n 2s, the nonexistence of solutions was roved in [2, 30, 31] in which the authors devised and used the Pohozaev tye identities. The Brezis-Nirenberg tye roblem was studied in [30] for s = 1/2 and [2] for 0 < s < 1. The Lemma s Hof and Maximum Princie was studied in [31]. An interesting interlay between the two oerators occur in case of eriodic solutions, or when the domain is the torus, where they coincide, see [11]. However in the case general the two oerators roduce very different behaviors of solutions, even when one focuses only on stable solutions, see e.g. Subsection 1.7 in [13]. Here we are interested in studying the roblem A s u = g(v A s v = f(u u = v = 0 on (1.2 where s (0, 1, f, g C(R, R n is a smooth bounded domain and A s denote sectral fractional Lalace oerators. By a weak solution of the system (1.2, we mean a coule (u, v Θ s ( Θ s (, satisfying A s/2 ua s/2 φ dx = g(vφ dx φ Θ s (

3 EJDE-2017/206 FRACTIONAL ELLIPTIC SYSTEMS 3 A s/2 ψa s/2 v dx = The main results of this aer are: f(uψ dx Theorem 1.1. Suose that 2 n < 4s, 0 < < F (r = r f(t dt. If there exist constants 0 { 2 if > 1 θ > if 1 and r 0 0 such that θf (r f(rr for all r r 0 and { o(r if > 1 f(r = o(r 1/ if 1 for r near 0. Then the system has a nontrivial weak solution. A s u = v A s v = f(u u = v = 0 on ψ Θ s (. 2s n 2s, f C(R, and set Note that if 2 n < 4s then n = 2 and s ( 1 2, 1 or n = 3 and s ( 3 4, 1. Theorem 1.2. Suose that n 4s, 0 < 1 and Then the system 0 < q < { n+4s n 4s if n > 4s 0 < q if n = 4s A s u = v A s v = u q u = v = 0 on (1.3 (1.4 (1.5 has a ositive weak solution. Moreover, if q < 1, then the roblem (1.5 admits a unique ositive weak solution. Remark 1.3. Suose that n 4s, 0 < q 1 and 0 < < n + 4s if n > 4s n 4s 0 < if n = 4s. Clearly we have a result analogous to the above theorem. When, q > 1, a riori bounds and existence of ositive solutions of (1.5 have been derived in [8, 20] rovided that, q satisfy q + 1 > n 2s. (1.6 n Remark 1.4. In the case when n 4s, the above Theorems cover the remaining cases below the critical hyerbola and when q 1. In the case when n > 4s, Figure 1 exemlifies the region that the above theorem covers.

4 4 E. J. F. LEITE EJDE-2017/206 Figure 1. Existence range of coules (, q when n > 4s. Remark 1.5. For such weak solutions, we rove an L estimate of Brezis-Kato tye and derive the regularity roerty of the weak solutions based on the results obtained in [31]. For s = 1, the roblem (1.5 and a number of its generalizations have been widely investigated in the literature, see for instance the survey [15] and references therein. Secifically, notions of sublinearity, suerlinearity and criticality (subcriticality, suercriticality have been introduced in [14, 24, 25, 28]. In fact, the behavior of (1.5 is sublinear when q < 1, suerlinear when q > 1 and critical (subcritical, suercritical when n 3 and (, q is on (below, above the hyerbola, known as critical hyerbola, q + 1 = n 2 n. When q = 1, its behavior is resonant and the corresonding eigenvalue roblem has been addressed in [26]. The sublinear case has been studied in [14] where the existence and uniqueness of ositive classical solution is roved. The suerlinearsubcritical case has been covered in the works [10], [16], [17] and [18] where the existence of at least one ositive classical solution is derived. Lastly, the nonexistence of ositive classical solutions has been established in [24] on star-shaed domains. When 0 < s < 1 and, q > 0, existence of ositive solutions of (1.5 for the restricted fractional Lalace oerator ( s have been derived in [20, 22] rovided that q 1 and (, q satisfies (1.6. Related systems have been studied with toological methods. We refer to the work [21] for systems involving different oerators ( s and ( t in each one of equations. The rest of aer is organized into five sections. In Section 2 we briefly recall some definitions and facts related to sectral fractional Lalace oerator. In Section

5 EJDE-2017/206 FRACTIONAL ELLIPTIC SYSTEMS 5 3, we rove the case > 1 of Theorem 1.1 by alying the Strongly Indefinite Functional Theorem of Li-Willem. Then, we rove the case 1 by using the mountain ass theorem of Ambrosetti-Rabinowitz. In Section 4, we rove the case q < 1 of Theorem 1.2 by using a direct minimization aroach, Hof lemma and maximum rinciles. Next we establish the remaining cases by using the mountain ass theorem. In Section 5, we establish regularity roerty of the weak solutions of system (1.4 based on the results obtained in [31]. Finally we establish the Brezis-Kato tye result and derive the regularity of solutions to ( Preliminaries In this section we briefly recall some definitions and facts related to sectral fractional Lalace oerator. The sectral fractional Lalace oerator A s is defined as follows. Let ϕ k be an eigenfunction of given by ϕ k = λ k ϕ k ϕ k = 0 on, (2.1 where λ k is the corresonding eigenvalue of ϕ k, 0 < λ 1 < λ 2 λ 3 λ k +. Then, {ϕ k } k=1 is an orthonormal basis of L2 ( satisfying ϕ j ϕ k dx = δ j,k. We define the oerator A s for any u C0 ( by A s u = λ s kξ k ϕ k, (2.2 where u = k=1 ξ k ϕ k and ξ k = k=1 k=1 uϕ k dx. This oerator is defined on a Hilbert sace Θ s ( = {u = u k ϕ k L 2 ( λ s k u k 2 < + } with values in its dual Θ s (. Thus the inner roduct of Θ s ( is given by u, v Θs ( = A s/2 ua s/2 v dx = ua s v dx = va s udx. We denote by Θ s the norm derived from this inner roduct. We remark that Θ s ( can be described as the comletion of the finite sums of the form f = c k ϕ k with resect to the dual norm f Θ s ( k=1 k=1 k=1 = (λ s k 1 c k 2 = A s/2 f 2 L = fa s f dx 2

6 6 E. J. F. LEITE EJDE-2017/206 and it is a sace of distributions. Moreover, the oerator A s is an isomorhism between Θ s ( and Θ s ( Θ s (, given by its action on the eigenfunctions. If u, v Θ s ( and f = A s u we have, after this isomorhism, f, v Θs ( Θ s ( = u, v Θs ( Θ s ( = λ s ku k v k. If it also haens that f L 2 (, then clearly we obtain f, v Θ s ( Θ s ( = fv dx. We have A s : Θ s ( Θ s ( can be written as A s f(x = G (x, yf(ydy, where G is the Green function of oerator A s (see [3, 19]. It is known that L 2 ( if s = 0 H s ( = H0( s if s (0, 1/2 Θ s ( = H 1/2 00 ( if s = 1/2 H0( s if s (1/2, 1] H s ( H0 1 ( if s (1, 2], where H 1/2 00 ( := {u H1/2 ( : u 2 (x d(x dx < + }, (see [18]. Observe that the injection Θ s ( H s ( is continuous. By the Sobolev imbedding theorem (see [12] we therefore have continuous imbeddings Θ s ( L +1 ( if + 1 2n 2n n 2s and these imbedding are comact if + 1 < n 2s for 0 < s < 2n. Also, (see [12], we have comact imbedding Θ s ( C(, if s n > 1 2, where C( is a Banach sace with the norm u C = su u. For 0 < r < 2 we have A s : Θ r ( Θ r 2s ( is an isomorhism (see [18]. Finally, by weak solutions, we mean the following: Let f L 2n n+2s (. Given the roblem A s u = f (2.3 u = 0 on, we say that a function u Θ s ( is a weak solution of (2.3 rovided A s/2 ua s/2 φ dx = fφ dx for all φ Θ s (. 3. Proof of Theorem 1.1 We organize the roof of Theorem 1.1 into two arts. We start by roving the existence of a weak solution in case > 1. k=1

7 EJDE-2017/206 FRACTIONAL ELLIPTIC SYSTEMS The case > 1. We define the roduct saces E α ( = Θ α ( Θ 2s α (. For 0 < α < 2s the sace E α ( is a Hilbert sace with inner roduct (u 1, v 1, (u 2, v 2 Eα ( = A α/2 u 1, A α/2 u 2 L2 ( + A s α/2 v 1, A s α/2 v 2 L2 (. We denote by E the norm derived from this inner roduct, i.e, (u, v E = ( u 2 Θ α + v 2 Θ 2s α 1/2. We also have A s : Θ α ( Θ α 2s ( is an isomorhism, see [18]. Hence ( 0 A s A s : E α ( Θ α Θ α 2s ( = E α ( 0 is an isometry. We consider the Lagrangian J (u, v = A s/2 ua s/2 v dx ( v +1 + F (u dx, (3.1 i.e., a strongly indefinite functional. The Hamiltonian is given by ( 1 H(u, v = + 1 v +1 + F (u dx. (3.2 The quadratic art can again be written as A(u, v = 1 2 L(u, v, (u, v E α ( = A α/2 ua s α/2 v dx = where ( 0 A s α L = A α s 0 is bounded and self-adjoint. Introducing the diagonals we have A s/2 ua s/2 v dx, E + = {(u, A α s u : u Θ α (} and E = {(u, A α s u : u Θ α (} E α ( = E + E. An orthonormal basis of E α ( is given by { 1 2 (λ α/2 k ϕ k, ±λ α/2 s k ϕ k : k = 1, 2,... }. The derivative of A(u, v defines a bilinear form B((u 1, v 1, (u 2, v 2 = A (u 1, v 1 (u 2, v 2 = L(u 1, v 1, (u 2, v 2 E α (, (3.3 where (u 1, v 1, (u 2, v 2 E α ( with A(u 1, v 1 = 1 2 B((u 1, v 1, (u 1, v 1 and B((u 1, v 1 +, (u 1, v 1 = 0. (3.4 We will give the choice of α in the following lemma. Lemma 3.1. Let 1 < < 2s/(n 2s. Then there exists a arameter 0 < α < 2s such that the following embeddings are continuous and comact, Θ 2s α ( L +1 ( and Θ α ( C(.

8 8 E. J. F. LEITE EJDE-2017/206 Proof. Note that Θ 2s α ( L q 2n ( comactly, if q < n 4s+2α and Θα ( C( comactly, if α n > 1 2, see [12]. We have + 1 < n n 2s. Thus if α > n 2, then + 1 < 2n n 4s+2α. For n = 2, we have s ( 1 2, 1. In this case the result is valid for all 1 < α < 2s. For n = 3, we have s (3/4, 1. In this case the result is valid for all 3 2 < α < 2s. This ends the roof. Remark 3.2. Note that α s > 0. Thus Θ α ( Θ s ( is comact. The functional J (u, v : E α ( R is strongly indefinite near zero, in the sense that there exist infinite dimensional subsaces E + and E with E + E = E α ( such that the functional is (near zero ositive definite on E + and negative definite on E. Li-Willem [23] rove the following general existence theorem for such situations, which can be alied in our case. Theorem 3.3 (Li-Willem, Let Φ : E R be a strongly indefinite C 1 - functional satisfying (i Φ has a local linking at the origin, i.e. for some r > 0: Φ(z 0 for z E +, z E r and Φ(z 0, for z E, z E r; (ii Φ mas bounded sets into bounded sets; (iii let E n + be any n-dimensional subsace of E + ; then Φ(z as z +, z E n + E ; (iv Φ satisfies the Palais-Smale condition (P S (Li-Willem [23] require a weaker (P S -condition, however, in our case the classical (P S condition will be satisfied. Then Φ has a nontrivial critical oint. We now verify that the functional defined in (3.1 satisfies the assumtions of this theorem. First, it is clear, with the choice of α (Lemma 3.1, that J (u, v is a C 1 -functional on E α (. We show that the condition (i of Theorem 3.3 is satisfied. It is easy to see that J (u, v has a local linking with resect to E + and E at the origin. Now the condition (ii of Theorem 3.3. Let B E α ( be a bounded set, i.e. u Θ α c, v Θ 2s α c, for all (u, v B. Then J (u, v A α/2 u L 2 A s α/2 v L 2 + v +1 dx + f(u dx u Θ α v Θ 2s α + c v +1 Θ 2s α + su{ f(u(x : x } C. We show that the condition (iii of Theorem 3.3 is satisfied. Let z k = z + k + z k E n + E denote a sequence with z k E +. Thus, z k may be written as z k = (u k, A α s u k + (w k, A α s w k, with u k Θ α n(, w k Θ α (, where Θ α n( denotes an n-dimensional subsace of Θ α (. Thus, the functional J (z k takes the form J (z k = A α/2 u k A s α/2 A α s u k dx A α/2 w k A s α/2 A α s w k dx

9 EJDE-2017/206 FRACTIONAL ELLIPTIC SYSTEMS = A α/2 u k 2 dx F (u k w k dx. A α s (u k w k +1 dx A α/2 w k 2 dx F (u k w k dx A α s (u k w k +1 dx Note that z k E if and only if A α/2 u k 2 dx + A α/2 w k 2 dx = u k 2 Θ + w k 2 α Θ. α Now, if (1 u k 2 Θ c, then w k 2 α Θ, and then J (z k ; α (2 u k 2 Θα +, then using the fact that α s > 0 and > 1 there exists c, c 1 and c 2 ositive constants such that ( +1 A α s (u k w k +1 dx c A α s (u k w k 2 2 dx c 1 u k w k +1 L 2 and F (u k + w k dx c 2 u k + w k +1 dx d c 1 u k + w k +1 L 2 and hence we obtain the estimate J (z k 1 ( 2 u k 2 Θ c α 1 u k w k +1 L + u 2 k + w k +1 L + d. 2 Since φ(t = t +1 is convex, we have 1 2 (φ(t + φ(r φ ( 1 2 (r + t, and hence J (z k 1 2 u k 2 Θ c 1 α 1 2 ( u k w k L 2 + u k + w k L d 1 2 u k 2 Θ c 1 α 1 2 u k +1 L + d. 2 Since on Θ α n( the norms u k Θ α and u k L 2 are equivalent (see [18], we conclude that also in this case J (z k. Finally, the condition (iv of Theorem 3.3. Let (z n E α ( denote a (P S- sequence, i.e. such that J (z n c, and (J (z n, η ɛ n η E, η E α (, and ɛ n 0. (3.5 Lemma 3.4. The (P S-sequence (z n is bounded in E α (. Proof. By (3.5 we have for z n = (u n, v n, J (u n, v n = A α/2 u n A s α/2 v n dx and where d vn +1 dx F (u n dx c J (u n, v n (ϕ, φ ɛ n (ϕ, φ E ɛ n ( ϕ Θ α + φ Θ 2s α, (3.6 J (u n, v n (ϕ, φ = A α/2 u n A s α/2 φ dx + A s α/2 v n A α/2 ϕ dx vnφ dx f(u n ϕ dx.

10 10 E. J. F. LEITE EJDE-2017/206 Choosing (ϕ, ψ = (u n, v n Θ α ( Θ 2s α ( we obtain by (3.6, 2 A α/2 u n A s α/2 v n dx vn +1 dx f(u n u n dx ɛ n ( u n Θ α + v n Θ 2s α and subtracting this from 2J (u n, v n we obtain, using assumtion (1.3 of Theorem 1.1 ( vn +1 dx + ( 1 2 f(u n u n dx C + ɛ n ( u n Θ α + v n θ Θ 2s α and thus vn +1 dx C + ɛ n ( u n Θ α + v n Θ 2s α, (3.7 f(u n u n dx C + ɛ n ( u n Θ α + v n Θ 2s α. (3.8 Choosing (ϕ, φ = (0, A α s u n Θ α ( Θ 2s α ( in (3.6 we obtain A α/2 u n 2 dx vna α s u n dx + ɛ n A α s u n Θ 2s α and hence by Hölder inequality, ( u n 2 Θ = α Aα/2 u n 2 L v 2 n +1 dx Noting that ( A α s u n +1 dx and using (3.7, we obtain ( 1 A α s u n dx +ɛ n u n Θ α. c A α s u n Θ 2s α = c A α/2 u n L 2 = c u n Θ α u n 2 Θ α [C + ɛ n( u n Θ α + v n Θ 2s α] /(+1 c u n Θ α + ɛ n u n Θ α ; therefore u n Θ α C + ɛ n ( u n Θ α + v n Θ 2s α /(+1. (3.9 As above we note that A s α v n Θ α (, and thus, choosing (ϕ, ψ = (A s α v n, 0 Θ α (Θ 2s α ( in (3.6 we obtain A s α/2 v n 2 dx f(u n A s α v n dx + ɛ n A s α v n Θ α A s α v n f(u n dx + ɛ n v n Θ α. Using that A s α v n Θ α = A s α/2 v n L 2 = v n Θ 2s α, and the fact that Θ α ( C( we then obtain, using (3.8, v n Θ α c f(u n dx + ɛ n = max f(t dx + f(u n u n dx + ɛ (3.10 n t s 0 [ u n s 0] C + ɛ n ( u n Θ α + v n Θ 2s α. Joining (3.9 and (3.10 we finally get [ u n >s 0] u n Θ α + v n Θ 2s α C + 2ɛ n ( u n Θ α + v n Θ 2s α. Thus, u n Θ α + v n Θ 2s α is bounded.

11 EJDE-2017/206 FRACTIONAL ELLIPTIC SYSTEMS 11 With this it is now ossible to comlete the roof of the (P S-condition: since u n Θ α is bounded, we find a weakly convergent subsequence u n u in Θ α (. Since the maings A α/2 : Θ α ( L 2 ( and A α/2 s : L 2 ( Θ 2s α ( are continuous isomorhisms, we obtain A α/2 (u n u 0 in L 2 ( and A α s (u n u 0 in Θ 2s α (. Since Θ 2s α ( L +1 ( comactly, we conclude that A α s (u n u 0 strongly in L +1 (. Similarly, we find a subsequence of (v n which is weakly convergent in Θ 2s α ( and such that vn is strongly convergent in L +1 (. Choosing (ϕ, φ = (0, A α s (u n u Θ α ( Θ 2s α ( in (3.6 we thus conclude A α/2 u n A α/2 (u n u dx vna α s (u n u dx + ɛ n A α s (u n u Θ 2s α. By the above considerations, the right-hand-side converges to 0, and thus A α/2 u n 2 dx A α/2 u 2 dx. Thus, u n u strongly in Θ α (. To obtain the strong convergence of (v n in Θ 2s α (, one roceeds similarly: as above, one finds a subsequence (v n converging weakly in Θ 2s α ( to v, and then A s α v n A s α v weakly in Θ α ( and A s α v n A s α v strongly in C(. Choosing in (3.5 (ϕ, φ = (A s α (v n v, 0, we obtain A s α/2 (v n va s α/2 v n dx f(u n A s α (v n vdx+ɛ n ( A s α (v n v Θ α. The first term on the right is estimated as A s α (v n v C and thus one concludes again that A s α/2 v n 2 dx f(u n dx 0, A s α/2 v 2 dx and hence also v n v strongly in Θ 2s α (. Thus, the conditions of Theorem 3.3 are satisfied; hence, we find a ositive critical oint (u, v for the functional J, which yields a weak solution to system ( The case 1: Variational setting. Suose that 1, n = 2 and s (1/2, 1 or n = 3 and s (3/4, 1. Thus, Θ 2s ( is comactly embedded in C(. Let be a smooth bounded oen subset of R n and 0 < s < 1. To motivate our formulation, assume that the coule (u, v of nontrivial functions is roughly a solution of (1.4. From the first equation, we have v = (A s u 1/. Plugging this equality into the second equation, we obtain A s (A s u 1/ = f(u u = 0 on. (3.11

12 12 E. J. F. LEITE EJDE-2017/206 The basic idea in trying to solve (3.11 is considering the functional Ψ : Θ 2s ( R defined by Ψ(u = A s u +1 dx F (udx. ( The Gateaux derivative of Ψ at u Θ 2s ( in the direction ϕ Θ 2s ( is Ψ (uϕ = A s u 1 1 A s ua s ϕ dx f(uϕ dx and thus, if f(u C(, the roblem A s v = f(u v = 0 on admits a unique nontrivial weak solution v Θ s (. Then, one easily checks that u is a weak solution of the roblem A s u = v u = 0 on. Remark 3.5. In short, starting from a critical oint u Θ 2s ( of Ψ, we have constructed a nontrivial weak solution (u, v Θ 2s ( Θ s ( of the roblem ( Existence of critical oints. In this subsection we rove the existence of a nontrivial weak solution of system (1.4. By Remark 3.5, it suffices to show the existence of a nonzero critical oint u Θ 2s ( of the functional Ψ. In this case 1, n = 2 and s ( 1 2, 1 or n = 3 and s ( 3 4, 1. Thus, Θ 2s ( is comactly embedded in C(. Then, the second term of the functional Ψ is defined if F is continuous, and no growth restriction on F is necessary. Since F is differentiable, the functional Ψ is a well-defined C 1 -functional on the sace Θ 2s (. The roof consists in alying the classical mountain ass theorem of Ambrosetti and Rabinowitz in our variational setting. We now show that Ψ has a local minimum at the origin. Ψ(u = A s u +1 dx F (udx + 1 c u C o ( u so that the origin u 0 = 0 is a local minimum oint. Next, let u 1 = tu, where t > 0 and u Θ 2s ( is a nonzero function. Then +1 t Ψ(u 1 A s u +1 dx t θ u θ C + d + 1 with θ > +1 (by assumtion, and thus Ψ(tu as t +. Finally, we show that Ψ fulfills the Palais-Smale condition (PS. Let (u k Θ 2s ( be a (PS-sequence, that is, Ψ(u k C 0, +1 C Ψ (u k ϕ ɛ k ϕ Θ 2s for all ϕ Θ 2s (, where ɛ k 0 as k +. We have C 0 + ɛ k u k Θ 2s,

13 EJDE-2017/206 FRACTIONAL ELLIPTIC SYSTEMS 13 θψ(u k Ψ (u k u k ( θ A s u k +1 dx θ ( θ A s u k +1 F (u k dx + +1 dx C0 δ u k Θ C 2s 0, f(u k u k dx and thus (u k is bounded in Θ 2s (. Thanks to the comactness of the embedding Θ 2s ( C(, one easily checks that (u k converges strongly in Θ 2s (. So, by the mountain ass theorem, we obtain a nonzero critical oint u Θ 2s (. This comletes the roof. 4. Proof of theorem 1.2 Note that n > 4s imlies that Θ 2s ( is continuously embedded in L 2n n 4s (. Now if n = 4s we have Θ 2s ( is comactly embedded in L r ( for all r > 1. Thus if u Θ 2s ( we have that u q L 2n n+2s ( for all q > 0. It suffices to rove the result for n > 4s, since the ideas involved in its roof are fairly similar when n = 4s Variational setting. Let be a smooth bounded oen subset of R n and 0 < s < 1.To motivate our formulation, assume that the coule (u, v of nonnegative functions is roughly a solution of (1.5. From the first equation, we have v = (A s u 1/. Plugging this equality into the second equation, we obtain A s (A s u 1/ = u q u 0 u = 0 on. (4.1 The basic idea for soving (4.1 is considering the functional Φ : Θ 2s ( R defined by Φ(u = A s u +1 1 dx (u + q+1 dx. ( q + 1 The Gateaux derivative of Φ at u Θ 2s ( in the direction ϕ Θ 2s ( is Φ (uϕ = A s u 1 1 A s ua s ϕ dx (u + q ϕ dx. In this case, Θ 2s ( is continuously embedded in L 2n n 4s (. Thus, if 0 < q n+2s we have u q L 2n n+2s (. Therefore, the roblem A s v = (u + q v = 0 on n 4s (4.3 admits a unique nonnegative weak solution v Θ s (. Now, if n+2s n 4s < q < n+4s n 4s, then u q Θ r 2s (, where 0 < r := n+4s (n 4sq 2 < s. Therefore (4.3 admits a unique nonnegative weak solution v Θ r (. Then, one easily checks that u is a weak solution of the roblem A s u = v u = 0 on.

14 14 E. J. F. LEITE EJDE-2017/206 In short, starting from a critical oint u Θ 2s ( of Φ, we have constructed a nonnegative weak solution { Θ 2s ( Θ s ( if 0 < q n+2s (u, v n 4s Θ 2s ( Θ r ( if n+2s n 4s < q < n+4s n 4s of roblem ( Existence for the case q < 1. We aly the direct method to the functional Φ on Θ 2s (. To show the coercivity of Φ, note that q + 1 < +1 because q < 1. Hence q < n+4s n 4s the embedding Θ2s ( L q+1 ( is continuous. So, for 1 there exist constants C 1, C 2 > 0 such that Φ(u = + 1 A s u +1 dx 1 q + 1 C u Θ C 2 2s q + 1 u q+1 Θ 2s +1 ( C1 = u Θ 2s + 1 C 2 (q + 1 u u q+1 dx +1 (q+1 Θ 2s for all u Θ 2s (. Therefore, Φ is lower bounded and coercive, that is, Φ(u + as u Θ 2s +. Let (u k Θ 2s ( be a minimizing sequence of Φ. It is clear that (u k is bounded in Θ 2s (, since Φ is coercive. So, module a subsequence, we have u k u 0 in Θ 2s (. Since Θ 2s ( is comactly embedded in L q+1 (, we have u k u 0 in L q+1 (. Here, again we use the fact that q + 1 < +1. Thus lim inf Φ(u k = lim inf n k + 1 As u k As u 0 L L +1 1 q + 1 u 0 q+1 L q+1 1 q + 1 u 0 q+1 L q+1 = Φ(u 0, so that u 0 minimizers Φ on Θ 2s (. We just need to guarantee that u 0 is nonzero. But, this fact is clearly true since Φ(ɛu 1 < 0 for any nonzero nonnegative function u 1 Θ 2s ( and ɛ > 0 small enough; that is, Φ(ɛu 1 = +1 ɛ + 1 A s u 1 +1 dx ɛ q+1 q + 1 for ɛ > 0 small enough. This ends the roof of existence. u 1 q+1 dx < Uniqueness for the case q < 1. The main tools in the roof of uniqueness are the strong maximum rincile and a Hof s lemma adated to fractional oerators. Let (u 1, v 1, (u 2, v 2 be two ositive solutions of (1.5. Define Γ = {γ (0, 1] : u 1 tu 2, v 1 tv 2 0 for all t [0, γ]}. From the strong maximum rincile and Hof s lemma (see [31], it follows that Γ is not emty. Let γ = su Γ and assume that γ < 1. Clearly, u 1 γ u 2, v 1 γ v 2 0. (4.4

15 EJDE-2017/206 FRACTIONAL ELLIPTIC SYSTEMS 15 By (4.4 and the integral reresentation in terms of the Green function G of A s (see [3, 19], we have u 1 (x = G (x, yv 1 (ydy G (x, yγ v 2 (ydy = γ G (x, yv 2 (ydy = γ u 2 (x for all x. In a similar way, one gets v 1 γ q v 2. Using the assumtion q < 1 and the fact that γ < 1, we derive A s (u 1 γ u 2 = v 1 γ v 2 (γq γ v 2 > 0 A s (v 1 γ v 2 = u q 1 γ u q 2 (γq γ u q 2 > 0 (4.5 So, by the strong maximum rincile, one has u 1 γ u 2, v 1 γ v 2 > 0. Then, by Hof s lemma, we have ν (u 1 γ u 2, ν (v 1 γ v 2 < 0 on, where ν is the unit outer normal in R n to, so that u 1 (γ +ɛu 2, v 1 (γ +ɛv 2 > 0 for ɛ > 0 small enough, contradicting the definition of γ. Therefore, γ 1 and, by (4.4, u 1 u 2, v 1 v 2 0. A similar reasoning also roduces u 2 u 1, v 2 v 1 0. This ends the roof of uniqueness Existence of critical oints in the case q > 1. From what we saw, it suffices to show the existence of a nonzero critical oint u Θ 2s ( of the functional Φ. The roof consists in alying the classical mountain ass theorem of Ambrosetti and Rabinowitz in our variational setting. We first assert that Φ has a local minimum in the origin. Note that 1 and q +1 > +1 because q > 1. Hence q < n+4s n 4s the embedding Θ 2s ( L q+1 ( is comact. Consider the set Γ := { u Θ 2s ( : u Θ 2s = ρ }. Then, on Γ, we have Φ(u = + 1 A s u +1 1 dx u q+1 dx q + 1 C u Θ C 2 2s q + 1 u q+1 > 0 = Φ(0 Θ = ρ +1 2s ( C1 + 1 C 2 q + 1 ρq+1 +1 for fixed ρ > 0 small enough, so that the origin u 0 = 0 is a local minimum oint. In articular, inf Γ Φ > 0 = Φ(u 0. Note that Γ is a closed subset of Θ 2s ( and decomoses Θ 2s ( into two connected comonents: { u Θ 2s ( : u Θ 2s < ρ } and { u Θ 2s ( : u Θ 2s > ρ }. Let u 1 = tu, where t > 0 and u Θ 2s ( is a nonzero nonnegative function. Since q > 1, we can choose t sufficiently large so that Φ(u 1 = +1 t A s u +1 t q+1 dx (u + q+1 dx < q + 1 It is clear that u 1 { u Θ 2s ( : u Θ 2s > ρ } and inf Γ Φ > max{φ(u 0, Φ(u 1 }, so that the mountain ass geometry is satisfied.

16 16 E. J. F. LEITE EJDE-2017/206 Finally, we show that Φ fulfills the Palais-Smale condition (PS. Let (u k Θ 2s ( be a (PS-sequence; that is, Φ(u k C 0, Φ (u k ϕ ɛ k ϕ Θ 2s for all ϕ Θ 2s (, where ɛ k 0 as k +. From these two inequalities and the assumtion q > 1, we deduce that C 0 + ɛ k u k Θ 2s (q + 1Φ(u k Φ (u k u k ( (q A s u k +1 dx +1 C u k Θ 2s and thus (u k is bounded in Θ 2s (. Thanks to the comactness of the embedding Θ 2s ( L q+1 (, one easily checks that (u k converges strongly in Θ 2s (. So, by the mountain ass theorem, we obtain a nonzero critical oint u Θ 2s (. This comletes the roof. 5. Regularity of solutions to systems 1.4 and 1.5 In this section, we establish regularity roerty of the weak solutions of system (1.4 based on the results obtained in [31]. Also we establish the Brezis-Kato tye result and derive the regularity of solutions to (1.5. Proosition 5.1. Let (u, v be a weak solution of the roblem (1.4. In the hyothesis of Theorem 1.1, we have (u, v L ( L ( and, moreover, (u, v C 1,β ( C β ( for some β (0, 1. Now if f is a C 1 function such that f(0 = 0 we have (u, v C 1,β ( C 1,β ( for some β (0, 1. Proof. In the case > 1 we find a solution (u, v Θ α ( Θ 2s α (. By choosing α (see Lemma 3.1, and by Sobolev imbedding theorem (see [12] we have u L (. Then f(u L (. Thus, by regularity result (see [31, Proosition 3.1] we have v C γ ( for some γ (0, 1. Hence γ + 2s > 1 and v C γ ( again by [31, Proosition 3.1] we have u C 1,γ+2s 1 (. Therefore (u, v C 1,β ( C β ( for some β (0, 1. Now if f is a C 1 function such that f(0 = 0 analogously we have (u, v C 1,γ+2s 1 ( C γ ( for some γ (0, 1. Then f(u C 2s (. Hence 2s + 2s > 1 again by [31, Proosition 3.1] we have v C 1,2s+2s 1 (. Therefore (u, v C 1,β ( C 1,β ( for some β (0, 1. In the case 1 we find a solution (u, v Θ 2s ( Θ s (. From Sobolev imbedding theorem (see [12] we have u L (. Analogous to the revious case, we have the result. Next we rove the L estimate of Brezis-Kato tye. Proosition 5.2. Let (u, v be a weak solution of the roblem (1.5. In the hyothesis of Theorem 1.2, we have (u, v L ( L ( and, moreover, (u, v C 1,β ( C 1,β ( for some β (0, 1.

17 EJDE-2017/206 FRACTIONAL ELLIPTIC SYSTEMS 17 Proof. In the case n > 4s we find a solution { Θ 2s ( Θ s ( (u, v Θ 2s ( Θ r ( if 0 < q n+2s n 4s if n+2s n 4s < q < n+4s n 4s, where 0 < r = n+4s (n 4sq 2 < s. We analyze searately two different cases deending on the values of q. Note that 0 < 1. For q > 1, we rewrite the roblem (1.5 as follows A s u = a(xv /2 A s v = b(xu u = v = 0 in R n \, (5.1 where a(x = v(x /2 and b(x = u(x q 1. Note that + 1 < n 2s and < 2n n 2r. By Sobolev embedding, Θs ( L +1 ( and Θ r ( L +1 (, so that a L 2(+1 (. Thus, for each fixed ɛ > 0, we can construct functions q ɛ L 2(+1 (, f ɛ L ( and a constant K ɛ > 0 such that a(xv(x /2 = q ɛ (xv(x /2 + f ɛ (x, q ɛ 2(+1 < ɛ, f ɛ L < K ɛ. L In fact, consider the set k = {x : a(x < k}, where k is chosen such that c k a(x 2(+1 dx < 1 2 ɛ 2(+1. This condition is clearly satisfied for k = k ɛ large enough. We now write { 1 q ɛ (x = m a(x for x k ɛ a(x for x c (5.2 k ɛ and f ɛ (x = (a(x q ɛ (x v(x /2. Then q ɛ (x 2(+1 dx = q ɛ (x 2(+1 dx + kɛ = ( 1 2(+1 m So, for m = m ɛ > ( 2 2(+1 ɛ < ( 1 2(+1 m a L c kɛ kɛ a(x 2(+1 dx + a(x 2(+1 kɛ 2(+1, we obtain 2n q ɛ (x 2(+1 dx c kɛ dx ɛ 2(+1. q ɛ 2(+1 < ɛ. L Note also that f ɛ (x = 0 for all x c k ɛ and, for this choice of m, f ɛ (x = (1 1 a(x 2 (1 1 kɛ 2 m ɛ m ɛ for all x kɛ. Therefore, f ɛ L (1 1 m ɛ k 2 ɛ := K ɛ. a(x 2(+1 dx

18 18 E. J. F. LEITE EJDE-2017/206 On the other hand, v(x = A s (bu(x, where b L q+1 q 1 (. Hence, u(x = A s [q ɛ (x(a s (bu(x /2 ] + A s f ɛ (x. By [8, Lemma 2.1], the claims (ii and (iv below follow readily and, by using Hölder s inequality, we also get the claims (i and (iii. Precisely, for fixed γ > 1, we have: (i The ma w b(xw is bounded from L γ ( to L β ( for 1 β = q 1 q γ ; (ii For 2s = n ( 1 β 2 θ, there exists a constant C > 0, deending on β and θ, such that (A s w /2 L θ C w /2 L β for all w L β (; (iii The ma w q ɛ (xw is bounded from L θ ( to L η ( with norm given by q ɛ L 2(+1, where θ 1 and η satisfies 1 η = 2( θ ; (iv For 2s = n ( 1 η 1 δ, the ma w A s w is bounded from L η ( to L δ (. From (i (iv, one easily checks that γ < δ and, in addition, u L δ A s[ q ɛ (x ( A s (bu /2 ] L δ + A s f ɛ L δ ( C q ɛ 2(+1 u /2 + f L L δ ɛ L δ. Using now the fact that 1, q ɛ L 2(+1 < ɛ and f ɛ L (, we deduce that u L δ C for some constant C > 0 indeendent of u. Proceeding inductively, we obtain u L δ ( for all δ 1. So, [8, Lemma 2.1] imlies that v L (. From this, and using [8, Lemma 2.1] again, we deduce that u L (. Then u q, v L (. Thus, by regularity result (see [31, Proosition 3.1] we have v, u C 2s (. Hence it holds that u q, v C 2s (. Again, we can aly regularity result to deduce that u C 4s (. Iteratively, we can raise the regularity so that (u, v C 1,β ( C 1,β ( for some β (0, 1. Now if q 1 write b(x = u(x q 2. In case n = 4s we find a solution (u, v Θ 2s ( Θ s (. Since Θ 2s ( is comactly embedded in L r ( for all r > 1, the result follows similarly. References [1] D. Alebaum; Lévy rocesses from robability to finance and quantum grous, Notices Amer. Math. Soc., 51 (2004, [2] B. Barrios, E. Colorado, A. de Pablo, U. Sánchez; On some critical roblems for the fractional Lalacian oerator, J. Differential Equations, 252 (2012, [3] M. Bonforte, J. L. Vázquez; A riori estimates for fractional nonlinear degenerate diffusion equations on bounded domains, Arch. Ration. Mech. Anal., 218 (2015, [4] C. Brändle, E. Colorado, A. de Pablo, U. Sánchez; A concave-convex ellitic roblem involving the fractional Lalacian, Proceedings of the Royal Society of Edinburgh, 143A (2013, [5] C. Bucur, E. Valdinoci; Nonlocal diffusion and alications, volume 20 of Lecture Notes of the Unione Matematica Italiana. Sringer, [Cham]; Unione Matematica Italiana, Bologna, xii+155. ISBN: ;

19 EJDE-2017/206 FRACTIONAL ELLIPTIC SYSTEMS 19 [6] X. Cabré, J. Tan; Positive solutions of nonlinear roblems involving the square root of the Lalacian, Advances in Mathematics, 224 (2010, [7] A. Caella, J. Dávila, L. Duaigne, Y. Sire; Regularity of Radial Extremal Solutions for Some Non-Local Semilinear Equations, Comm. Partial Differential Equations, 36 (2011, [8] W. Choi; On strongly indefinite systems involving the fractional Lalacian, Nonlinear Anal., 120 (2015, [9] W. Choi, S. Kim, K. Lee; Asymtotic behavior of solutions for nonlinear ellitic roblems with the fractional Lalacian, J. Funct. Anal., 266 (2014, [10] Ph. Clément, D. G. de Figueiredo, E. Mitidieri; Positive solutions of semilinear ellitic systems, Comm. Partial Differential Equations, 17 (1992, [11] R. de la Llave, E. Valdinoci; A generalization of Aubry-Mather theory to artial differential equations and seudo-differential equations, Ann. Inst. H. Poincaré Anal. Non Linéaire 26 (2009, no. 4, [12] F. Demengel, G. Demengel; Functional Saces for the Theory of Ellitic Partial Differential Equations, (Universitext-Sringer (2012. [13] S. Diierro, N. Soave, E. Valdinoci; On stable solutions of boundary reaction-diffusion equations and alications to nonlocal roblems with Neumann data, Indiana Univ. Math. J., htts: // [14] P. Felmer, S. Martínez; Existence and uniqueness of ositive solutions to certain differential systems, Adv. Differential Equations, 4 (1998, [15] D. G. de Figueiredo; Semilinear ellitic systems, Nonl. Funct. Anal. Al. Diff. Eq., World Sci. Publishing, River Edge (1998, [16] D.G. de Figueiredo, P. Felmer; On suerquadratic ellitic systems, Trans. Amer. Math. Soc. 343 (1994, [17] D. G. Figueiredo, B. Ruf; Ellitic systems with nonlinearities of arbitrary growth, Mediterr. J. Math., 1 (2004, [18] J. Hulshof, R. van der Vorst; Differential Systems with Strongly Indefinite Variational Structure, J. Funct. Anal., 114 (1993, [19] T. Jakubowski; The estimates for the Green function in Lischitz domains for the symmetric stable rocesses, Probab. Math. Statist. 22 (2002, [20] E. Leite; On strongly indefinite systems involving fractional ellitic oerators, arxiv: , To aear in Far East Journal of Alied Mathematics. [21] E. Leite, M. Montenegro; A riori bounds and ositive solutions for non-variational fractional ellitic systems, arxiv: , To aear in Differential and Integral Equations. [22] E. J. F. Leite, M. Montenegro; On ositive viscosity solutions of fractional Lane-Emden systems, arxiv: , [23] S. Li, M. Willem; Alications of local linking to critical oint theory, J. Math. Anal. Al., 189 (1995, [24] E. Mitidieri; A Rellich tye identity and alications, Comm. Partial Differential Equations, 18 (1993, [25] E. Mitidieri; Nonexistence of ositive solutions of semilinear ellitic systems in R N, Differential and Integral Equations, 9 (1996, [26] M. Montenegro; The construction of rincial sectral curves for Lane-Emden systems and alications, Ann. Scuola Norm. Su. Pisa Cl. Sci., 29 (2000, [27] R. Musina, A. I. Nazarov; On fractional Lalacians, Comm. Partial Differential Equations, 39 (2014, [28] J. Serrin, H. Zou; Existence of ositive entire solutions of ellitic Hamiltonian systems, Comm. Partial Differential Equations, 23 (1998, [29] R. Servadei, E. Valdinoci; On the sectrum of two different fractional oerators, Proc. Roy. Soc. Edinburgh, Sect. A 144 (2014, [30] J. Tan; The Brezis-Nirenberg tye roblem involving the square root of the Lalacian, Calc. Var. Partial Differential Equations, 42 (2011, [31] J. Tan; Positive solutions for non local ellitic roblems, Discrete and Continuous Dynamical Systems, 33 (2013, [32] A. Xia, J. Yang; Regularity of nonlinear equations for fractional Lalacian, Proc. Amer. Math. Soc., 141 (2013,

20 20 E. J. F. LEITE EJDE-2017/206 Edir Junior Ferreira Leite Deartamento de Matemática, Universidade Federal de Viçosa, CCE, , Viçosa, MG, Brazil address: