Screening masses in a neutral two-flavor color superconductor

Transcription

1 PHYSICAL REVIEW D, VOLUME 7, 943 Screening masses in a neutral two-flavor color suerconductor Mei Huang* and Igor A. Shovovy Institut für Theoretische Physi, J.W. Goethe-Universität, D-654 Franurt/Main, Germany (Received 6 Setember 4; ublished 17 November 4) The Debye and Meissner screening masses of the gluons and the hoton in neutral and -equilibrated dense two-flavor quar matter are calculated. The results are resented in a general form that can be used in gaed as well as galess color suerconducting hases. The results for the magnetic screening masses indicate that the system develos a chromomagnetic instability. Possible consequences of the instability are discussed. DOI: 1.113/PhysRevD I. INTRODUCTION From the time when the quars were redicted [1], their nature has remained rather elusive. The reason is that direct exerimental studies of quars are very limited. Quars do not exist in vacuum as free articles. Under normal conditions, they are always confined inside hadrons. The underlying theory of strong interactions quantum chromodynamics (QCD) redicts that quars should become deconfined at very high temeratures and/or very high densities [,3]. Unfortunately, it is very difficult to achieve sufficiently high densities and/or temeratures in laboratory. Very high temeratures existed in the early Universe during the first few microseconds of its evolution [4]. Nowadays, somewhat similar conditions, although at considerably smaller scales and for much shorter eriods of time, are reeatedly recreated in the so-called little bangs at the heavy ion colliders in CERN and at Broohaven. Sufficiently high densities may exist in the resent Universe inside central regions of comact stars. Recently, this ossibility attracted a lot of attention when it was suggested that various color suerconducting hases with rather large values of gas in their quasiarticle energy sectra could aear at densities that exist inside stars [5 8]. If this turns out to be true, this would be of rime imortance. The resence of large gas in the energy sectra can ossibly be inferred from a detailed analysis of the observational data. This would rovide a confirmation of the existence of new (quar) states of matter inside comact stars. In theoretical studies of dense quar matter, it should be areciated that matter in the bul of stars is neutral and -equilibrated. Under such conditions, the chemical otentials of different quars should satisfy nontrivial relations. These, in turn, affect the airing dynamics *Electronic address: huang@th.hysi.uni-franfurt.de; on leave of absence from Physics Deartment, Tsinghua University, Beijing 184, China Electronic address: shovovy@th.hysi.uni-franfurt.de; on leave of absence from Bogolyubov Institute for Theoretical Physics, 3143, Kiev, Uraine PACS numbers: 1.38.Aw, 1.38.Mh, 6.6.+c between quars which is reflected in a secific choice of the ground state of matter. For examle, it was argued in Ref. [9] that a mixture of the two-flavor color suerconducting (SC) hase and normal strange quars is less favorable than the color-flavor-loced (CFL) hase after the charge neutrality condition is enforced. A similar conclusion was also reached in Ref. [1]. Assuming that the constituent medium modified mass of the strange quar is large (i.e., larger than the corresonding strange quar chemical otential), it was shown recently that neutral two-flavor quar matter in equilibrium can have another rather unusual ground state called the galess two-flavor color suerconductor (gsc) [11]. While the symmetry in the gsc ground state is the same as that in the conventional SC hase, the sectrum of the fermionic quasiarticles is different. In articular, two out of four gaed quasiarticles of the conventional SC hase become galess in the gsc hase. In addition, the number densities of the airing quars in the gsc hase are not equal at zero temerature [11]. For examle, the density of red (green) u quars is different from the density of green (red) down quars. The existence of galess color suerconducting hases was confirmed in Refs. [1,13], and generalized to nonzero temeratures in Refs. [14,15]. In addition, it was also shown that a galess CFL (gcfl) hase could aear in neutral strange quar matter [16,17]. At nonzero temerature, the gcfl hase and several other new hases (e.g., the so-called dsc and usc hases) were studied in Refs. [18,19]. If the surface tension is sufficiently small, as suggested in Ref. [], the mixed hase comosed of the SC hase and the normal quar hase will be more favored [1]. It was also suggested that a nonrelativistic analogue of galess suerconducting hases could aear in a traed gas of cold fermionic atoms [ 5]. (Note that an alternative ground state for the atomic system, similar to the quar mixed hases in Refs. [,1,6], was roosed in Ref. [7].) While the basic roerties of galess color suerconducting hases have been established in Refs. [11 19], there is robably much more to be learned about these hases in the near future. In this aer, we give a detailed =4=7(9)=943(4)$ The American Physical Society

2 MEI HUANG AND IGOR A. SHOVKOVY PHYSICAL REVIEW D derivation of the gluon and hoton screening masses in neutral dense quar matter (the results were briefly resented in Ref. [8]). We consider the general case of twoflavor quar matter, covering both the gaed and the galess SC hases. The gluon screening roerties in the case of the ideal SC hase (i.e., without a mismatch between the Fermi momenta of different quars) were considered in detail in Refs. [9,3]. Some general features of the gluon screening in the gsc hase were conjectured in Refs. [14,31] without erforming the calculation. As we shall see below, the actual results for the Debye screening masses are in general agreement with the conjecture in Refs. [14,31]. The Meissner (magnetic) screening roerties, however, are very different [8]. The calculations in this aer indicate a chromomagnetic instability in neutral dense quar matter for a range of arameters in the model. As we shall argue in Sec. VIII, this instability may lead to a gluon condensation in dense quar matter. This aer is organized as follows. The linear resonse theory is briefly reviewed in Sec. II. After that, in Sec. III, we introduce the model and set u the main notation. There, we also resent the general exression for the quar roagator in the color suerconducting ground state of neutral dense quar matter. In Sec. IV, we briefly discuss the general exression for the olarization tensor in dense quar matter. In Sec. V, we study the olarization tensor AB for A; B 1; ; 3 and derive the corresonding exressions for the Debye and Meissner screening masses. We show that, in accordance with the symmetry breaing attern, there is no Meissner effect (i.e., no Higgs mechanism) in this sector of the gauge theory. The olarization tensor AB for A; B 8; 9 (i.e., the 8th gluon and the hoton) is discussed in Sec. VI. There we derive the Debye and Meissner screening masses and briefly discuss their roerties. Also, the mixing between the 8th gluon and the hoton is discussed. In Sec. VII, we study the olarization tensor AB for A; B 4; 5; 6; 7. The results for the Debye and Meissner screening masses are resented. As we show, the Debye screening mass is given by a rather simle exression that naturally interolates between the limits of the normal hase and the ideal SC hase. The Meissner mass, on the other hand, has an unexected roerty. Its value squared is negative in a range of arameters, indicating a chromomagnetic instability in dense quar matter. The discussion of the main results is given in Sec. VIII. Our findings are summarized in Sec. IX. Several aendices at the end contain useful formulas and some details of the calculation. II. LINEAR RESPONSE THEORY AND POLARIZATION TENSOR In order to resent a self-contained discussion of the screening roerties of dense quar matter, we start this aer with a brief introduction into the linear resonse theory and a discussion of the hysical meaning of the olarization tensor in a gauge theory. The advanced reader, therefore, may si this section and go directly to Sec. III. The resonse of matter to an external erturbation is the main source of nowledge about roerties of matter. The linear resonse theory is the simlest framewor that is often used to calculate such a resonse. In alication to quar matter, for examle, one studies a resonse of the system to an external source J A x. The source is couled to the quantum gauge field. The corresonding interaction art of the action reads S J Z d 4 xa A; xj A x i Z d 4 x Z d 4 ya A; xd 1 AB x ya B; y; (1) where A B; y is the classical field associated with the external source J A x, andd 1 AB is the inverse free gluon roagator. Because of the resence of the external source, the exectation value of the gauge field becomes nonzero. In the linear resonse theory, it is given by the Kubo s formula [3], ha A xi i Z d 4 yd AB x yj ;B y; () where D AB x y is the retarded gluon roagator. In momentum sace, this relation taes the following form: ha A Pi id AB PJ ;B P; (3) where P ; is the energy-momentum four-vector. By maing use of this result, it is instructive to derive an exression for the induced current. It is given by J A;ind P J A;tot PJ A P id 1 AB PD 1 AB PhA B; Pi AB PhA B; Pi; (4) where AB P is the gluon self-energy (or the gluon olarization tensor). By definition, this is the one-article irreducible art of the gluon two-oint function. The structure of this function is constrained by the gauge symmetry. To see this we consider the Slavnov-Taylor identity for the full gluon roagator [33]. The exlicit form of this identity deends on a secific gauge fixing. In the covariant gauge, for examle, it reads P P id AB P P P id AB P 1 ; (5) where is the gauge fixing arameter. In vacuum, where Lorentz symmetry is not broen, this relation imlies that the gluon self-energy is transverse, i.e., P AB P. Because of this constraint, the tensor structure of AB P in vacuum is fixed unambiguously, 943-

3 SCREENING MASSES IN A NEUTRAL TWO-FLAVOR... PHYSICAL REVIEW D AB P AB P g P P P ; (6) where the metric tensor is defined as g diag1; 1; 1; 1. At nonzero temeratures and/or densities, the Lorentz symmetry is broen down to its subgrou of satial rotations SO(3). Then, the tensor structure of the gluon self-energy can have a more general form, AB; P g u u H AB u u K AB L AB u u M AB ; (7) where u 1; ; ; is a timelie four-vector that secifies the rest frame of the quar system and ; is the momentum three-vector with the absolute value jj. The comonent functions H AB, K AB, L AB, and M AB are functions of and. Now, the Slavnov- Taylor identity in the covariant gauge taes the form KL K L MM ; (8) (for simlicity, the suerscrits AB were omitted here). It is easy to chec that this is less restrictive than the transversality condition, P P, required in vacuum. Indeed, the transversality is equivalent to the following set of two relations between the comonent functions: L K (9a) M K: (9b) While these are sufficient conditions to fulfil the Slavnov- Taylor identity in Eq. (8), they are not the necessary conditions in a non-abelian gauge theory when the Lorentz symmetry is broen. (It should be emhasized, however, that these are the necessary conditions in Abelian gauge theories at nonzero temeratures and/or densities [3].) In this aer, we study the olarization tensor AB P in the case of dense quar matter which ermits color suerconductivity. In articular, we discuss how the structure of the olarization tensor is affected by the (galess) color suerconductivity. It is usually said that suerconductivity is a result of a gauge symmetry breaing. This common misleading statement may suggest that the olarization tensor AB P does not need to satisfy the Slavnov-Taylor identity (5). In fact, this is not the case because a local (gauge) symmetry can never be truly broen [34]. In ractice, when doing secific calculations in gauge theories, one always breas the local symmetry by a gauge fixing. As a result, it is only a global symmetry that may remain unbroen after the gauge choice is made. For examle, in the covariant gauge which we discussed above, the global color symmetry of the QCD action is left unbroen. Then, in a color suerconducting hase of quar matter, this global symmetry is broen. In such a descrition, the Goldstone theorem requires the aearance of an aroriate number of the Nambu-Goldstone bosons (cf., collective excitations in Ref. [35]). Obviously, the aearance of these additional degrees of freedom is an artifact of the gauge fixing. Nevertheless, their inclusion in the analysis is imortant in order to insure that the general requirements of the gauge symmetry, such as the Slavnov-Taylor identity (5), are fulfilled [36]. Having said this, one should areciate that the Nambu-Goldstone bosons in question are not the hysical degrees of freedom. This conclusion is easy to reach by noticing that there exists a gauge, namely, the so-called unitary gauge, in which these bosons can be comletely eliminated. In a way, their role is similar to the role of the Faddeev-Poov ghosts [37]. While both tyes of fields are unhysical, they are necessary for a consistent descrition of a gauge theory. Now let us further discuss the hysical meaning of the olarization tensor. From a relation similar to that in Eq. (4), it is clear that, in an Abelian theory such as QED, this tensor is directly related to an observable quantity, namely, to the induced current. The corresonding current is a gauge invariant quantity in an Abelian theory. In contrast, the induced current in a non-abelian theory is not a gauge invariant quantity. Then, the hysical meaning of the olarization tensor is not so clear. In site of this difficulty, we shall use the same interretation of the olarization tensor in quar matter as in an Abelian theory. In this aer, we study static large-distance, electric and magnetic, screening roerties of quar matter. These describe the resonse of the system to a static erturbation from color/electric charges and currents. The static limit means that. In this case, the only nontrivial comonents of the olarization tensor will be H and K which deend only on. Note that ; K; ij ; g ij i j H: (1a) (1b) Let us denote the values of the two nontrivial comonent functions in the limit! (large distances) as follows: m D lim K;! (11a) m M lim H:! (11b) The quantities m D and m M are the Debye and Meissner screening masses, resectively. It can be shown, see, for examle, Ref. [3,38], that the quantity m D determines 943-3

4 MEI HUANG AND IGOR A. SHOVKOVY PHYSICAL REVIEW D the large-distance behavior of the screened otential created by a static color/electric charge, i.e., Vr exm D r. By maing use of the analogy with solid state hysics systems, we say that a system is a metal when m D is nonzero, and it is an insulator when m D is zero. The quantity m M, when nonzero, determines the large-distance falloff of the (chromo-)magnetic field inside a color suerconductor, i.e., Brexm M r. Obviously, nonzero m M is an indication of the Meissner effect. In the normal hase, on the other hand, the value of m M is vanishing. III. QUARK PROPAGATOR In this aer, we continue the study of dense two-flavor quar matter constrained by the conditions of the charge neutrality and the equilibrium [9 19,39 41]. The use of henomenological Nambu-Jona-Lasinio (NJL)-tye models has roved to be very convenient in such studies. The NJL model can be thought of as a low-energy theory of QCD in which (massive) gluons, as indeendent degrees of freedom, are integrated out. The gluons could be reintroduced bac by gauging the color symmetry in the NJL model, roviding a semirigorous framewor for studying the effect of the Cooer airing on the hysical roerties of gluons. In order to study the gluon screening roerties in dense quar matter, we need to now the quar couling to the gauge fields. This is determined by the quadratic art of the quar Lagrangian density L m ^ g A a T a e A Q ; (1) where T a and Q are the generators of SU3 c and U1 em gauge grous. The couling constants of the strong interactions and of the electromagnetism are denoted by g and e, resectively. The u and down quar masses are assumed to be the same (m u m d m). The quar sinor field has the following color-flavor structure: i ur ug ub dr dg db 1 ; (13) C A where i u; d is the flavor index and r; g; b is the color index. In equilibrium, the matrix of chemical otentials in the color-flavor sace ^ is given in terms of the quar chemical otential (note that B 3 is the baryon chemical otential), the chemical otential for the electrical charge e, and the color chemical otential 8, ij ij e Q ij 8 ij T 8 : (14) 3 In QCD the color chemical otential 8 comes as a result of a nonzero exectation value of the 8th gluon field induced by the Cooer airing [4]. Its absolute value is small because it is suressed arametrically by the quar chemical otential, 8 =. The exlicit exressions for nontrivial elements of matrix ^ read ur ug 3 e 1 3 8; dr dg 1 3 e 1 3 8; ub 3 e 3 8; db 1 3 e 3 8: (15a) (15b) (15c) (15d) To study color suerconducting hases, it is convenient to introduce the following 8N c N f -comonent Nambu- Gorov sinors: ; C ; ; (16) where C C T is the charge-conjugate sinor, and C i is the charge-conjugation matrix. In this basis, the quadratic art of the quar Lagrangian density L becomes L S 1 A A ^ A ; (17) where the exlicit form of vertex ^ A is ^ A diag A ; A diagg T A ; g TA T for A a 1;...; 8; diage Q; e Q for A 9 hoton: (18) In momentum sace, the inverse free quar roagator S 1 reads S K 1 G K1 G : (19) K1 The exlicit form for the diagonal elements is G 1 E ur E ur E dg E dg P E ub E ub 3 P E db E db P 4 ; () with the notation E i E i and E m. The four rojectors P I (with I 1;...; 4) in the six C P

5 SCREENING MASSES IN A NEUTRAL TWO-FLAVOR... PHYSICAL REVIEW D dimensional color-flavor sace are defined as follows: P 1 ij b b iu ju ; (1a) P ij b b id jd ; (1b) P 3 ij b b iu ju ; (1c) P 4 ij b b id jd : (1d) It is not difficult to see that P 1 rojects out the red u and the green u quars, and P rojects out the red down and the green down quars. The rojectors P 3 and P 4 roject out the blue u and the blue down quars, resectively. In Eq. (), we also introduced the energy rojectors, 1 m 1 : () E These rojectors satisfy the following relations [43]: where ~ 1 ~ ; 5 5 ~ ; 1 m E (3a) (3b) (4) is an alternative set of energy rojectors. In the chiral limit, the two sets of rojectors in Eqs. () and (4) coincide. The full quar roagator in a color suerconducting hase taes the following form: with SK 1 G K1 G K1 i b " 5 ; y i b " 5 ; ; (5) (6a) (6b) where b is the antisymmetric tensor in the color subsace sanned by the red and green colors, while " is the antisymmetric tensor in the flavor sace. The value of the ga arameter is determined from an aroriate ga equation, while the values of the chemical otentials e and 8 are determined from charge neutrality conditions [9 11,13,14,39 41]. The exlicit form of the ga equation and the neutrality conditions is not imortant for the uroses of this aer. From Eq. (5), we obtain where S G G G G 1 G 1 ; G G ; ; (7) (8a) (8b) with the free quar roagators G ~ E ~ ~ ur E P 1 ur E dg ~ ~ E P dg E ~ ub E P 3 ub ~ E ~ db E P 4 ; (9) db obtained from Eq. (). By maing use of the following relations: "P 1 " P ; b P 1 b P 1 ; (3a) "P " P 1 ; b P b P ; (3b) "P 3 " P 4 ; b P 3 b ; (3c) "P 4 " P 3 ; b P 4 b ; (3d) one can derive an exlicit form of the Nambu-Gorov comonents of the full roagator, G X4 I1 G I P I ; 1 b P 1 "P 1 b P "P 1 : The exlicit form of the functions G I and IJ reads (31a) (31b) G 1 E dg E ~ G E ur E ~ G 3 1 E ~ 1 bu E bu G 4 1 E bd E dg E E ur E ~ ; ~ 1 E ~ ; bd ~ ; ~ ; (3a) (3b) (3c) (3d) and 943-5

6 MEI HUANG AND IGOR A. SHOVKOVY PHYSICAL REVIEW D i 1 E 5 ~ 1 E 5 ~ ; (33a) 1 i 1 E 5 ~ 1 E 5 ~ ; (33b) where the following notation was used: E E ; (34a) q E ; E ; (34b) ur dg dg ur ug dr dr ug e ; (34c) e : (34d) In the following sections, we use the full quar roagator in Eq. (7) to construct the olarization tensor for the gauge fields. IV. POLARIZATION TENSOR IN DENSE QUARK MATTER In dense quar matter, screening effects lay a very imortant role at length scales larger than the average distance between quars. In the normal hase, for examle, the main effects are the Debye screening and the Landau daming. These are the roerties that can be extracted from the behavior of the olarization tensor. The olarization tensor in dense matter is given by the socalled hard dense loo (HDL) aroximation [44,45]. This aroximation results from taing into account only the dominant one-loo quar contribution in which the internal quar momenta are hard (i.e., tyical momenta are of order ). The density of quar states with hard momenta is roortional to (i.e., the density of states at the Fermi surface). Because of this large density of states, the quar HDL contribution is large comared to the contributions from the gluon and the ghost loos. Therefore, the gluon and the ghost contributions are not included in the HDL aroximation. In the SC/gSC hase of dense quar matter, the olarization tensor is given aroximately by the following one-loo exression [9,3]: AB P 1 T V X Tr D;c;f;NG ^ A SK ^ B SK P; (35) K where the trace runs over the Dirac, color, flavor, and Nambu-Goov indices. The gluon art (A; B 1;...; 8) of this olarization tensor reduces to the standard HDL result in the normal hase ( ), ;ab ab u u Q HDL P 4 s 1 Q 1 g u u Q u ; (36) Q u where s g =4, and Qx 1 Z 1 d x i" x i" x ln 1 x 1 x 1 i jxj1 x : (37) In the static limit ( ), we obtain Q 1, and the olarization tensor becomes ;ab HDL ; 4 s ab u u : (38) This gluon olarization tensor describes the static screening of quar color charges at large distances in the normal hase of dense quar matter. By comaring with Eqs. (1) and (11), we derive the corresonding exression for the Debye screening mass, m D 4 s : (39) As it should be, the Meissner screening mass is zero in the normal hase. V. GLUONS WITH A 1; ; 3 In this section, we start with the screening roerties of the A 1; ; 3 gluons. These are the gluons of the unbroen SU c subgrou which coule only to the red and green quars. The corresonding exression for the 943-6

7 SCREENING MASSES IN A NEUTRAL TWO-FLAVOR... PHYSICAL REVIEW D olarization tensor is diagonal, AB P AB 11 P. After erforming the traces over the color, flavor and Nambu- Gorov indices, we arrive at 11 P T X Z d 3 g 4 n 3 Tr D G 1 K G 1 K G 1 K G 1 K G K G K G K G K 1 K 1 K 1 K 1 K 1 K 1 K 1 K 1 K ; (4) where T is the temerature and K K P. Here we use the imaginary time formalism, and the energy integration is relaced by the sum over the fermionic Matsubara frequencies! n Tn 1. The exlicit exressions for the comonents G I and IJ of the quar roagator are given in Eqs. (3) and (33). After the summation over the Matsubara frequencies, we obtain the result in the following form: 11 P Z d 3 s 3 C11 C T C 11 C T C 11 C T C 11 C T C 1 C 1 U C 1 C 1 U C 1 C 1 U C 1 C 1 U : (41) In this exression, we introduced the following notation for the two tyes of Dirac traces: T e 1 e Tr D ~ e 1 ~ e ; U e 1 e Tr D 5 ~ e 1 5 ~ e ; (4a) (4b) with e 1 ;e. To leading order in 1=, the results for these traces are given in Eqs. (A1) (A4) in Aendix A. The exressions for the coefficient functions C IJ at zero and nonzero temeratures are given in Aendix B 1. We write the integral over the three-momentum in Eq. (41) as Z d 3 Z d 4 Z 1 1 d Z d ; (43) where is the olar angle and is the cosine of the angle between the three-momenta and. After erforming the integral over the olar angle, see Eqs. (A5) and (A6), the corresonding traces in the integrand can be relaced by the following angular averaged exressions: T! u u u u 1 g u u 13 ; (44a) T!1 g u u 13 ; (44b) U!1 g u u 13 ; (44c) U!u u 1 g u u 1 3 u u : (44d) Now we would lie to note that there are two different tyes of coefficient functions in Eq. (41). The coefficients C 11, C, C 1, andc 1 originate from article-hole and antiarticle-antiarticle loos. In the leading order aroximation, we dro the antiarticle-antiarticle contributions to the olarization tensor. These are suressed by an inverse ower of the quar chemical otential. In addition, in the static long-wavelength limit ( and! ), the article-hole contributions simlify considerably. The aroximate exressions read C 11 C C 1 C 1 4E ; 3 1 E ; E ; E ; E E ; E 4E ; ; (45a) 4E ; 3 1 E ; E ; E ; E E ; E 4E ; ; (45b) 4E 1 E ; 3 ; E ; 4E ; : (45c) 943-7

8 MEI HUANG AND IGOR A. SHOVKOVY PHYSICAL REVIEW D Here we neglected the corrections due to nonzero quar Z masses. This is justified if the shift of the quar Fermi d C 11; 1 " momenta due to such masses is small, i.e., m 1 ; (49a) =. We also used the following relations valid when! : Z d C 1;1 1 " 1 ; (49b) j j ; (46a) E E E ; (46b) E E ; E ; E ; ; E E ; E ; E ; E E E ; : (46c) (46d) As is easy to chec, the terms with the -and functions in Eq. (45) contribute only in the gsc hase when < (this also includes the normal hase as a limiting case with ). Now, the coefficients C 11, C, C 1, andc 1 are made of article-antiarticle loos only. Their contributions to the olarization tensor are not negligible although they are formally suressed by an inverse ower of the quar chemical otential. In fact, the corresonding contributions are ultraviolet divergent. The divergences are removed by subtractions of the corresonding vacuum terms. One can also chec that a nonzero ga arameter ( ) leads only to a small correction to these article-antiarticle contributions. Thus, we aroximate the corresonding coefficient functions as follows: C 11 C C 1 C 1 : j j j j j j 1 ; (47a) (47b) While the traces in Eq. (44) deend on the angular coordinate, the aroximate coefficient functions C IJ in Eqs. (45) and (47) do not. Therefore, the corresonding angular integration in Eq. (41) can be easily erformed, Z 1 1 Z 1 1 Z 1 1 Z 1 1 dt 4u u 4 3 g u u ; dt 8 3 g u u ; du 8 3 g u u ; du 4u u 4 3 g u u : (48a) (48b) (48c) (48d) The results on the right-hand side are indeendent of the momentum. Therefore, in order to derive the final exression for the olarization tensor, we need to calculate only the following momentum integrals: Z d C 11; 1 ; (49c) Z d C 1;1 : (49d) The details of the calculation are given in Aendix C 1. Note that, in the SC hase when <, the contributions from the normal (C 11; ) and abnormal (C 1;1 ) quar loos are equal, see Eqs. (49a) and (49b). The additional contributions in the gsc hase have equal absolute values but differ in sign. As for the article-antiarticle quar loo, only the normal (C 11; ) contribution is nonvanishing. By substituting the results in Eqs. (48) and (49) into Eq. (41), we derive the following exression for the olarization tensor 11 4 s 3 11 : Z d 3C 11 C 1 u u C 11 C 11 C 1 4 s g u u u u : (5) Only the -comonent of this olarization tensor is nonzero. This comonent determines the Debye screening mass, m D; s : (51) In the SC hase ( >), where there are no galess excitations charged with resect to the unbroen SU c subgrou, this screening mass is identically zero [9]. In contrast, it is nonzero in the gsc hase ( <). In fact, its value is roortional to the density of the galess quasiarticle states. As for the Meissner screening mass, its value vanishes in the SC hase as well as in the gsc hase, m M;11 1 lim g ij i j! ij 11; : (5) VI. 8TH GLUON, PHOTON AND THE GLUON- PHOTON MIXING In the SC/gSC hase, the diquar condensate breas the gauge symmetry SU3 c U1 em down to the SU c ~U1 em subgrou. The structure of the condensate in the ground is such that the 8th gluon and the 943-8

9 SCREENING MASSES IN A NEUTRAL TWO-FLAVOR... PHYSICAL REVIEW D hoton mix. The new medium fields are given by the following linear combinations of the vacuum fields [46]: ~A 8 A 8 cos A sin ; (53a) ~A A cos A 8 sin ; (53b) where the mixing angle is determined from the structure of the condensate by using simle grou-theoretical arguments, s 3 cos s ; (54a) 3 s s sin : (54b) 3 s The generator of the medium (low-energy) electromagnetism ~U1 em is defined as follows: ~Q Q 1 3 T 8. The 8th gluon can robe the Cooer-aired red and green quars, as well as the unaired blue quars. After the traces over the color, the flavor, and the Nambu-Gorov indices are erformed, the olarization tensor for the 8th gluon can be exressed as 88 P 1 3 ~ 88 P 3 88;b P; ~ 88 P T X Z d 3 g 4 n 3 Tr D G 1 K G 1 K G 1 K G 1 K G K G K G K G K 1 K 1 K 1 K 1 K 1 K 1 K 1 K 1 K ; 88;b P T X Z d 3 g 4 n 3 Tr D G 3 K G 3 K G 3 K G 3 K G 4 K G 4 K G 4 K G 4 K : (55a) (55b) (55c) Note that the contribution ~ 88 of aired quars, u to a sign in front of the contribution from the abnormal quar loos, is the same as the exression for 11 in Eq. (4). Therefore, by following the same stes as in the revious section, we easily derive the result ~ 88 4 Z s d 3C 11 C 1 3 u u C 11 C 11 C 1 g u u 4 s u u 4 s 3 " 1 g u u ; (56) [c.f. Eq. (5)]. The unaired blue quars give the standard normal hase HDL contribution 88;bP HDL P, see Eq. (36). In the static ( ) long-wavelength (! ) limit, this leads to the following result: 88;b 4 s u u : (57) Thus, the final result for the olarization tensor of the 8th gluon reads 88 4 s u u 4 " s 1 9 g u u : (58) Because of the symmetry considerations, one should exect a nontrivial gluon-hoton mixing in the ground state. So, we roceed to the calculation of the hoton olarization tensor. The general exression for this tensor is 943-9

10 MEI HUANG AND IGOR A. SHOVKOVY PHYSICAL REVIEW D P 1 ~ 99 P1 99;b P; (59a) ~ 99 P T X Z d 3 e 9 n 3 Tr Df4 G 1 K G 1 K G 1 K G 1 K G K G K G K G K 1 K 1 K 1 K 1 K 1 K 1 K 1 K 1 K g; (59b) 99;b P T X Z d 3 e 9 n 3 Tr Df4 G 3 K G 3 K G 3 K G 3 K G 4 K G 4 K G 4 K G 4 K g: (59c) Let us first start with the contribution of the unaired blue quars 99;bP. This is roortional to the standard HDL result, 99;b P 1=9 s HDL P where e =4 is the fine structure constant and HDLP is given in Eq. (36). At and!, wearriveat 99;b 4 9 u u : (6) To calculate the contribution of the aired quars ~ 99 P, we use the same aroach as in the revious section. After erforming the Matsubara frequency summation, we obtain the following reresentation: Z ~ d 3 99 P C11 C T 4C 11 C T 4C 11 C T 4C 11 C T C 1 C 1 U C 1 C 1 U C 1 C 1 U C 1 C 1 U : (61) By substituting the results in Eqs. (48) and (49) into Eq. (61), we derive the following exression for the olarization tensor ~ 99 in the static long-wavelength limit: Z ~ d 35C 11 4C 1 7 u u 5C 11 1C 11 4C 1 g u u " " u u 8 1 g u u : (6) 9 7 By combining the results in Eqs. (6) and (6), we obtain the exression for the hoton olarization tensor " u u " g u u : (63) Now, we calculate the gluon-hoton mixed comonents of the olarization tensor. The corresonding exression is given by 943-1

11 SCREENING MASSES IN A NEUTRAL TWO-FLAVOR... PHYSICAL REVIEW D P 98 P 1 ~ 89 P1 89;bP; (64a) ~ 89 89;b P egt 3 X Z d 3 3 n 3 Tr Df G 1 K G 1 K G 1 K G 1 K G K G K G K G K 1 K 1 K 1 K 1 K 1 K 1 K 1 K 1 K g; Z d 3 3 Tr Df G 3 K G 3 K G 3 K G 3 K G 4 K G 4 K P egt 3 X 3 n G 4 K G 4 K g: The contribution of the unaired blue quars is roortional to the HDL exression, s 89;b P HDLP; (65) 3 3 s with HDL P defined in Eq. (36). At and!, we obtain s 89;b 8 3 u u : (66) 3 To calculate the contribution of the aired quars, ~ 89, we first erform the Matsubara frequency summation. The result is ~ 89 P 4 s 3 Z d C11 C T C 11 C T C 11 C T C 11 C T C 1 C 1 U C 1 C 1 U C 1 C 1 U C 1 C 1 U : (67) (64b) (64c) By substituting the results in Eqs. (48) and (49) into Eq. (67), we get the following exression for the olarization tensor ~ 89 in the static long-wavelength limit: ~ s Z d 3C 11 C 1 u u 3 C 11 C 11 C 1 g u u 8 s 3 u u 8 s " 1 g u u ; (68) [c.f. Eq. (56)]. By substituting the results in Eqs. (66) and (68) into Eq. (64a), we obtain the exression for the mixed comonents of the olarization tensor " s g u u : (69) It is imortant to emhasize that the exlicit result for the electrical -comonent of this mixing gluon-hoton olarization tensor is vanishing at and!. This is similar to the ideal SC case considered in Ref. [3]. (Because of this, one should be careful when interreting the results for the Debye screening masses in a different basis of gauge fields [47].) The exressions for the Debye screening masses read m D;88 4 s ; (7a) m 8 3 D; 1 3 : (7b) In order to obtain the Meissner screening masses, we first derive all comonents of the olarization tensor that san the sace of the 8th gluon and the hoton. At and!, the corresonding nonzero comonents, denoted as m M;AB, read m M;88 4! s 1 ; (71a) 9! m 4 M; 1 ; (71b) 7! m 4 s M;8 9 1 ; (71c) 3 and m M;8 m M;8. The mixing disaears in the basis of the fields in Eq. (53). The Meissner screening mass for!

12 MEI HUANG AND IGOR A. SHOVKOVY PHYSICAL REVIEW D the ~8th gluon field is m M;~8 ~8 43! s 44 P T Z d 3 g 8 3 Tr D G 3 K G 1 K 1 ; (7) 7 G 1 K G 3 K G 4 K G K and the Meissner screening mass for the medium hoton G K G 4 K G 3 K G 1 K ~ is vanishing, m M;~ ~. This is consistent with the G 1 K G 3 K G 4 K G K absence of the Meissner effect for the unbroen ~U1 em. G As is easy to see from Eq. (7), the medium modified K G 4 K : (73) ~8th gluon has a chromomagnetic instability in the galess [Note that SC hase. This is because the Meissner screening mass 44 P 55 P 66 P 77 P.] Aart from the diagonal elements, there are also nonzero squared is negative when < = < 1. off-diagonal elements, VII. GLUONS WITH A 4; 5; 6; 7 45 P 54 P 67 P 76 After erforming the traces over the color, the flavor, P and the Nambu-Gorov indices, the diagonal comonents i ^ P; (74) of the olarization tensor ABP with A B 4; 5; 6; 7 have the form with ^ P g T Z d Tr D G 3 K G 1 K G 1 K G 3 K G 4 K G K G K G 4 K G 3 K G 1 K G 1 K G 3 K G 4 K G K G K G 4 K : (75) The off-diagonal comonents of the gluon self-energy in Eq. (74) are nonzero in general. The hysical gluon fields in the SC/gSC hase are the following linear combinations: ~A 4;5 A 4 ia 5 = and ~A 6;7 A 6 ia 7 = [9]. These new fields, ~A 4;6 and ~A 7;5, describe two airs of massive vector articles with well defined electromagnetic charges, ~Q 1. The comonents of the olarization tensor in the new basis read ~ 44 P ~ g T 4 66 P 44 P ^ P Z d 3 3 Tr D G 3 K G 1 K G 1 K G 3 K G 4 K G K G K G 4 K ; (76a) and ~ 55 P ~ 77 P 44 P ^ P g T Z d Tr D G 1 K G 3 K G 3 K G 1 K G K G 4 K G 4 K G K : (76b) After Matsubara frequency summation, the olarization tensors can be written in the following form: ~ 44 P Z d 3 s T C 44 T C 44 T C 44 T ; ~ 55 P s 3 C44 Z d 3 3 C55 T C 55 T C 55 T C 55 T : The exlicit exressions for the coefficient functions C 44;55 and C 44;55 at zero and nonzero temeratures are given in Aendix B. Here we quote only the aroximate zero temerature results at and!. The article-hole contributions are C 44;55 1 E ; (77a) (77b) E ; E E ; E E bu; E bd; E b; E b; E ; E E ; E ; E E b; b; E ; E ; (78) b; E ; 943-1

13 SCREENING MASSES IN A NEUTRAL TWO-FLAVOR... PHYSICAL REVIEW D and the article-antiarticle contributions are C 44;55 j j j j b j j 1 ; (79) where E b; E b and b 8. The vacuum contribution was subtracted in Eq. (79) in order to remove ultraviolet divergences in the olarization tensor. To roceed, we need to calculate the following two tyes of momentum integrals (for the details of the calculation see Aendix C ): " Z d C 44; ln and " 4 ln for 8! ; (8a) Z d C 44;55 : (8b) By substituting the results in Eqs. (48), (8a), and (8b) into Eq. (77), we derive exressions for the olarization tensors in the static long-wavelength limit: ~ 44;55 ~ 44 ~ 55 s 3 " 4 s s " Z d 3C 44 u u C 44 C 44 g u u 4 8 ln ln ln u u ln g u u : (81) In the neutral SC/gSC hase of quar matter, the value of the color chemical otential 8 is small [14]. Therefore, a good aroximation for the above olarization tensors is obtained by taing the limit 8!, ~ 44 ~ 55 " s u u " s g u u : (8) 3 Therefore, the corresonding Debye and Meissner screening masses are m D;44 m D;55 " s ; (83a) m M;44 m M;55 " s : (83b) 3 It is clear from Eq. (83b) that the corresonding four gluons have chromomagnetic (lasma-tye) instabilities in the galess SC hase ( < = < 1), as well as in the gaed SC hase when 1 < = <.Thisis indicated by a negative value of the Meissner screening mass squared. VIII. DISCUSSION The results of our calculation for the screening masses in neutral two-flavor quar matter are summarized in Table I. There we list the squared values of the Debye and Meissner screening masses in units of m g 4 s =

14 MEI HUANG AND IGOR A. SHOVKOVY PHYSICAL REVIEW D TABLE I. The Debye and Meissner screening masses for gauge bosons in neutral two-flavor quar matter. By definition, m g 4 s =3. m D;x =m g m M;x =m g 3 x 11; ; 33 x 44; 55; 66; x x 8; 8 1 x 3 3 s s s 1 9 x ~8 ~8 s 3 s s 9 s 9 s 1 x ~8 ~; ~ ~8 3 3 s 3 s 1 3 s 1 3 x ~ ~ 9 3 s 1 A. Debye screening masses Let us first discuss the results for the Debye screening masses. The values of m D;x =m g with x 11; 44; 88 as functions of the dimensionless ratio = are shown grahically in Fig. 1 (we do not show the Debye screening mass for the hoton whose value is suressed by the fine structure constant). The vanishing value of the ratio = corresonds to the normal hase of quar matter. In this limit, as is easy to chec, our results for the Debye screening masses coincide with the nown results in Refs. [44,45]. Also, in the other limiting case, = 1, our results coincide with those in the ideal SC hase [9,3]. As before, by the ideal SC hase of quar matter, we mean the SC hase without any mismatch m D /m g / µ FIG. 1 (color online). Squared values of the Debye screening masses as functions of the dimensionless ratio = for the gluons with A 1; ; 3 (solid line), for the gluons with A 4; 5; 6; 7 (long-dashed line), and for the ~8th gluon (short-dashed line). By definition, m g 4 s =3. between the Fermi momenta of the u and down quars, i.e.,. We see that the value of the Debye screening mass for the gluons of the unbroen SU c gauge subgrou is vanishing in the SC hase (= > 1). This result can be understood in the same way as in the ideal SC case [9]. It reflects the fact that there are no galess quasiarticles charged with resect to the SU c subgrou in the ground state. In the gsc hase, on the other hand, there are galess charged quasiarticles around the two effective Fermi momenta eff F [11,14]. The densities of states at the corresonding Fermi surfaces are [14] dn de : (84) As one can chec from Table I, the squared value of the Debye screening mass m D;11 is roortional to the sum of these densities of the galess states (u to higher order corrections). Turning these arguments around, one could obtain the result for the Debye screening mass without calculating the olarization tensor. Indeed, one can write [31] m dn D;11 dn ; (85) de de and determine the overall coefficient s = by matching the exression on the right-hand side with the nown exression in the normal hase (= ). It is interesting to consider m D;11 in the limit when the ga aroaches from the side of the galess hase. As one can see, the formal value of m D;11 goes to infinity as s = 1 = when!. Thisisthe

15 SCREENING MASSES IN A NEUTRAL TWO-FLAVOR... PHYSICAL REVIEW D consequence of having a quadratic disersion relation for galess quasiarticles E = when!. However, the infinitely large value of the Debye mass has little hysical meaning. This is because the distance scales, at which the corresonding screening is set in, also become infinite in the limit!. From Table I, we see that there is no mixing between the Debye screening masses of the 8th gluon and of the vacuum hoton. It is nown, however, that the hysical modes of the corresonding gauge bosons in the SC/ gsc hase are the linear combinations given in Eq. (53) which are different from the vacuum fields. Moreover, one of the linear combinations, see Eq. (53b), describes the medium modified hoton of the unbroen electromagnetism. One might thin that the absence of mixing between the electric screening masses is in conflict with the gauge invariance of the SC/gSC ground state with resect to ~U1 em. However, the two roagating modes of the lowenergy hoton ~ of ~U1 em have transverse olarizations and, therefore, should come from the magnetic sector. The third, electrical mode of ~ is not massless. This latter decoules from the low-energy theory and its resence is irrelevant for the gauge invariance with resect to ~U1 em. m M /m g /δµ FIG. (color online). Squared values of the Meissner screening masses as functions of the dimensionless ratio = for the gluons with A 1; ; 3 (solid line), for the gluons with A 4; 5; 6; 7 (long-dashed line), and for the ~8th gluon (short-dashed line). By definition, m g 4 s =3. B. Meissner screening masses Now we discuss the results for the Meissner screening masses. The values of m M;x =m g with x 11; 44; 88 as functions of the dimensionless ratio = are shown grahically in Fig. (the screening mass for the hoton, suressed by the fine structure constant, is not shown). As it should be, the results in the two limiting cases, i.e., = and = 1, coincide with the results in the normal hase (i.e., no Meissner effect) and with the results in the ideal SC hase [9,3], resectively. In agreement with the grou-theoretical arguments, the Meissner masses of the three gluons of the unbroen SU c subgrou are vanishing in both the galess and the gaed SC hases, see the solid line in Fig.. The vanishing Meissner masses come as a result of the exact cancellation between the diamagnetic article-antiarticle and the aramagnetic article-hole contributions. The results in Table I show that the Meissner screening masses in the subsace of the 8th gluon and the vacuum hoton have a nontrivial mixing. As we saw in Sec. VI, the mixing disaears after switching to the descrition in terms of the hysical modes, defined in Eq. (53). Moreover, the Meissner screening mass of the medium hoton is vanishing. This is consistent with the absence of the Meissner effect for the unbroen electromagnetism. The most interesting results of our calculation are the exressions for the Meissner screening masses of the five gluons that corresond to the broen generators of the SU3 c grou. We find that the squared values of these masses are negative (i.e., the masses themselves are imaginary) in some regions of arameters. In articular, we obtain m M;x < for x 44; 55; 66; 77 when < = <, and m < when < = < 1. M;~8 ~8 The standard interretation of such a result is the existence of a chromomagnetic (lasma-tye) instability in the corresonding hases of matter [8]. While the instability connected with the ~8th gluon aears only in the gsc hase ( < = < 1), the instability connected with the other four gluons also develos in the gaed SC hase when 1 < = <. This suggests that the galess suerconductivity itself is not the reason for the instability. It is interesting to note, however, that the critical value of the mismatch, above which the instability starts to develo, is given by c =. This haens to coincide with the value of the mismatch at which the SC hase becomes metastable when the neutrality condition is not enforced in quar matter (i.e., when the value of is treated as a free arameter and the Coulomb effects are ignored). Here, on the other hand, we consider neutral quar matter in the same way as in Refs. [11,14]. It is natural to suggest that the instability, indicated by negative values of the Meissner screening masses squared, may result in some tye of a gluon condensation. Indeed, in terms of the gluon effective otential, the aearance of a negative gluon mass squared could be viewed as considering a false vacuum that corresonds to a local maximum of the otential at ha a i. Inthiscase,the true vacuum should be given by the global minimum of the gluon otential. It would be natural then if the minimum corresonds to another (nonzero) exectation value of the gluon field, i.e., ha a i for a 4; 5; 6; 7, or for a 8. Note that we do not exclude the ossibility that the new stable ground state has a condensate that breas the rotational symmetry of the system. In fact, this might be the

16 MEI HUANG AND IGOR A. SHOVKOVY PHYSICAL REVIEW D most natural outcome of the gluon condensation if its mechanism is the same, e.g., as in Ref. [48,49]. The ossibility of breaing the rotational symmetry is also suggested by the fact the it is the magnetic comonents of gluons A a i rather than the electric gluons Aa that drive the instability. The exectation that the rotational symmetry is broen in the ground state may also hint at the ossibility of a state with deformed quar Fermi surfaces which was roosed in Ref. [5]. It may haen that not only the rotational symmetry but also the translational symmetry is broen in the true ground state of neutral two-flavor quar matter. This would be the case when the diquar condensate is inhomogeneous lie, for examle, in the crystalline hase [51], or lie in the Abriosov s vortex lattice hase [5]. Regarding the crystalline hase (also nown as the Larin-Ovchinniov-Fulde-Ferrell hase [53]) in twoflavor quar matter, it might be aroriate to mention that this hase is claimed to aear recisely when = <. It is not clear, however, how a first order transition from the SC hase to the crystalline hase at = can be triggered by vanishingly small tachyonic masses of gluons. We would lie to emhasize here that the gluon-tye instability, indicated by negative values of the Meissner screening masses squared, has nothing to do (at least, directly) with the so-called Sarma instability [54]. It was shown for the first time in Ref. [11], and then confirmed in Refs. [1 19] that the Sarma instability in the effective otential for the order arameter is removed when the neutrality condition is imosed, or when the airing interaction has a secific momentum structure and the ratio of the densities of the airing fermions is et fixed [5]. One may still argue that, desite the absence of the Sarma instability in the effective otential, there exists another tye of an instability, discussed in Ref. [55]. There it was suggested that aramagnetic currents should be sontaneously induced in the galess hase. In fact, we thin that the instability connected with the ~8th gluon in the gsc hase could indeed be of this tye. One may find some similarity between the chromomagnetic instability found in this aer and the instabilities that have been discussed in Refs. [56 6] in alication to a state of matter roduced in heavy ion collisions. It should be emhasized, however, that the quar distribution functions are comletely isotroic in momentum sace in the case of dense quar matter studied in this aer which is in contrast to a tyical situation in Ref. [56 6]. This fact suggests that the mechanism of the instability seen here is more subtle. In assing, we would lie to mention that there might exist some analogy between the instability found in this aer and the so-called aramagnetic suerconductivity discussed in condensed matter hysics [61]. If these henomena are related indeed, it would be natural to exect that the instability of neutral dense quar matter is resolved through a sontaneous chromomagnetization. Again, the breadown of the rotational symmetry in this case would seem inevitable. IX. CONCLUSION In this aer, we calculated the Debye and Meissner screening masses for the gluons and the hoton in the case of neutral, -equilibrated two-flavor dense quar matter. A general form of our result allows to use it in the gaed hase as well as in the galess color suerconducting hase by a simle change of the magnitude of the diquar airing. This latter determines the ratio of the ga arameter and the mismatch of the quar Fermi momenta, i.e., =. The qualitative deendence of this ratio on the diquar couling constant in the case of neutral quar matter was studied in Refs. [11,14]. It was shown that the ground state corresonds to the normal hase at wea couling, i.e., =. At some intermediate values of the diquar couling, the ground state is the gsc hase. The ratio = is less than 1 in such a galess hase. Finally, at large couling, the ground state is the SC hase, and the ratio = is larger than 1. One could also chec that this ratio = increases monotonically with increasing the couling. The results for the Debye and the Meissner screening masses in this aer give a natural interolation between the nown values in the normal hase [44,45] and in the ideal SC hase [9,3]. The most imortant result of our calculation is that the exressions for the Meissner screening masses of the five gluons, corresonding to the five broen generators of the SU3 c grou, are imaginary. This is interreted as an indication of a chromomagnetic instability in the corresonding hases of quar matter. It remains to be clarified the consequences of such an instability, and the nature of the true ground state in neutral two-flavor quar matter. In the future, it would be very interesting to investigate whether a chromomagnetic instability also develos in the gcfl hase [16,17,19], where the low-energy quasiarticle sectrum resembles the sectrum in the gsc hase. ACKNOWLEDGMENTS The authors than M. Buballa, M. Forbes, T. Hatsuda, C. Kouvaris, J. Lenaghan, M. S. Li, V. Miransy, S. Mrówczyńsi, R. Pisarsi, K. Rajagoal, A. Rebhan, S. Reddy, P. Reuter, D. Rische, P. Romatsche, T. Schäfer, A. Schmitt, D. Son, M. Stricland, M. Tachibana, D. N. Vosresensy, and Q. Wang for interesting discussions. I. A. S. is grateful to the INT at the University of Washington in Seattle for its hositality. The wor of M. H. was suorted by the Alexander von Humboldt- Foundation and the NSFC under Grant Nos and