Nematicity and quantum paramagnetism in FeSe

Nematicity and quantum paramagnetism in FeSe Fa Wang 1,, Steven A. Kivelson 3 & Dung-Hai Lee 4,5, 1 International Center for Quantum Materials, School of Physics, Peking University, Beijing 100871, China. Collaborative Innovation Center of Quantum Matter, Beijing 100871, China 3 Department of Physics, Stanford University, Stanford, California 94305, USA. 4 Department of Physics, University of California, Berkeley, CA 9470, USA. 5 Materials Sciences Division, Lawrence Berkeley National Laboratory, Berkeley, CA 9470, USA. 1 The Spin-1 Néel-Nematic Transition as a double Spin-1/ Néel-VBS Transition It has been proposed that a Landau-forbidden continuous quantum phase transition between Néel order and valence bond solid can happen in spin-1/ systems on square lattice. This transition can be described by a non-linear sigma model (NLσM) with a Wess-Zumino-Witten(WZW) term and certain anisotropy terms 1,. The action of this model reads S 1 [ ˆϕ] =S O(3) C4v [ ˆϕ] 3 πi 8π dud xdτ ϵ abcdf ϕ a x ϕ b y ϕ c τ ϕ d u ϕ f. (1) Here the 5-component superspin ˆϕ (n x,n y,n z,v x,v y ) consists of the Néel order parameters n =(n x,n y,n z ) ( 1) x+y S (x,y), and the two columnar VBS order parameters v =(v x,v y ) email: dunghai@berkeley.edu NATURE PHYSICS www.nature.com/naturephysics 1

where v x ( 1) x (S (x,y) S (x+1,y) S (x,y) S (x 1,y) ), (a) v y ( 1) y (S (x,y) S (x,y+1) S (x,y) S (x,y 1) ). (b) u is the auxiliary dimension for defining the WZW term. S O(3) C4v is the non-topological part of the non-linear sigma model action with O(3) C 4v symmetry. This action includes the stiffness terms such as ( d 1 xdτ g n µ n + 1 g v µ v ). In addition it contains anisotropy terms that favor the Néel order parameters over the VBS order parameters. Moreover, among the VBS order parameters there are terms that favor the columnar VBS over the plaquette VBS order. Examples of such anisotropy terms include (ϕ 4 + ϕ 5) (where > 0) and U (ϕ 4 ϕ 5) (where U<0). In particular, > 0 insures that the low energy physics is described by the fluctuations of the Néel order parameters. Spin-1 can be viewed as two spin-1/s coupled by strong ferromagnetic(fm) interaction. We thus consider two copies of the action equation (1) labeled by superscripts (1) and (), S 1 [ ˆϕ (1), ˆϕ () ]=S1[ ˆϕ (1) ]+S1[ ˆϕ () ]+ d xdτ ( J n n (1) n () + J v v (1) v ()). (3) We assume FM coupling between the Néel order parameters (J n < 0) and AFM coupling between the VBS order parameters (J v > 0) so that the low energy configurations have n (1) = n () n and v (1) = v () v. The action in terms of ˆϕ =(n, v) will be similar to equation (1) but with a doubled WZW term. Note however v = 1 (v(1) v () ) cannot be directly measured in spin-1 systems, because all NATURE PHYSICS www.nature.com/naturephysics

SUPPLEMENTARY INFORMATION physical observables must be symmetric with respect to exchange of the two spin-1/ moments. Define two physical order parameters, v 1 = v(1) x v x () v (1) v x + vy y v y (), v = v(1) x v y () + v () v x + vy x v y (1). (4) v 1 carries lattice momentum (0, 0), belongs to the B 1 representation of C 4v (changes sign under 4-fold rotation, but has no sign change under principal axis reflection), and corresponds to the nematic order parameter Ω 4 defined in the main text (for example, the parent Hamiltonian ground state X in the main text corresponds to v x (1) = v x () 0 and v y (1) = v y () = 0 and thus v 1 > 0). v has lattice momentum (π, π), is the B representation of C 4v (changes sign under both 4-fold rotation and principal axis reflection around a lattice site), and corresponds to certain superpositions of plaquette valence bond solid order (for example, v < 0 may correspond to either v (1) x = v () y > 0 with plaquette singlets centered at (x +1/, y +1/), or v (1) y = v () x < 0 with plaquette singlets centered at (x 1/, y 1/) for integer x, y). If we parametrize v by (v x,v y )=(v cos θ, v sin θ), then v =(v 1,v )=(v cos θ, v sin θ). Note that v 1 µ v v µ v 1 =v µ θ =(v x µ v y v y µ v x ), µ v 1 ν v µ v ν v 1 =v ( µ v ν θ µ θ ν v)=( µ v x ν v y µ v y ν v x ). Therefore the WZW term in action S 1 in terms of ˆϕ (n, v )=(n x,n y,n z,v 1,v ) has a halved coefficient compared to that in terms of (n, v). The action then becomes S 1 [ ˆϕ (1), ˆϕ () ] S 1 [ ˆϕ ]= S O(3) Z Z [ ˆϕ 3 ] πi dud xdτ ϵ abcdf ϕ 8π a x ϕ b y ϕ c τ ϕ d u ϕ f. (5) The action S O(3) Z Z [ ˆϕ ] is derived from S O(3) C4v [ ˆϕ] while taking equation (4) into account. It has O(3) Z Z anisotropy induced by the additional anisotropy terms such as 4 ϕ 4 and 5 ϕ 5 where 4,5 > 0. NATURE PHYSICS www.nature.com/naturephysics 3

Assume 5 4 > 0. Consider the configuration ˆϕ (u, x, y, τ) = (ˆΩ(x,y,τ) sin(u), cos(u) ), where ˆΩ is the 4-component real vector field in Eq. (6) of the main text. When u =0this is a uniform space-time configuration, and when u = π/ this will be a low energy space-time configuration with ϕ 5 =0. Integrate over u from 0 to π/, the WZW term becomes 3 π/ πi d xdτ ϵ abcd Ω 8π a x Ω b y Ω c τ Ω d du ( sin 5 u sin 3 u cos u) 0 3 = πi d xdτ ϵ abcd Ω 8π a x Ω b y Ω c τ Ω d ( /3) π = i d xdτ ϵ abcd Ω π a x Ω b y Ω c τ Ω d In the above the ( sin 5 u) and ( sin 3 u cos u) terms are respectively from abcdf = abcd5 and abcdf =5bcda terms in the WZW model (terms with index 5 at other positions vanish). This result is exactly the Θ-term in equation (6) of the main text with Θ=π. Thus the final effective action is given by equation (6) of the main text. (Note that due to the anisotropy term V (Ω 4) the stiffness constant for the first three components of ˆΩ will be different from that of Ω 4 at low energies and long wavelengths.) In the above discussion we have made the assumptions that J n < 0 and J v > 0 in equation (3), and 5 4 > 0 in equation (5). Here we briefly comment on several other possibilities. (i). If J n < 0 and J v < 0, the low energy configurations in equation (3) will be n (1) = n () n and v (1) = v () v, this theory would describe the phase transition from Néel AFM order (n) to columnar VBS order (v) with a WZW term similar to that of equation (1) but with doubled coefficient. 4 NATURE PHYSICS www.nature.com/naturephysics

SUPPLEMENTARY INFORMATION (ii). If J n > 0, the low energy configurations in equation (3) would have n (1) = n () n, then n is the director of the uniaxial spin-nematic order (uniaxial spin-nematic states have n S =0 ). This theory would describe the phase transition between ferro-spin-nematic order and columnar VBS order (if J v < 0) or a nematic quantum paramagnet (if J v > 0). However the WZW term would be absent, so we expect this phase transition to be of first-order. (iii). If J n < 0 and J v > 0 as we assumed in equation (3), but 4 5 > 0 in equation (5). This theory would describe the phase transition between Néel order and plaquette VBS order. The effects of explicit C 4 C symmetry breaking field. To explicitly break the C 4 symmetry to C, we can introduce a Z Zeeman field S S + h d xdτ Ω 4 (x, y, τ), (6) which breaks the degeneracy between the two types of monopole in Eq.(7) of the main text, making the absolute value of the Feynman amplitude associated them different hence they do not cancel. Now tunneling events involving unit changes in the skyrmion number are allowed, which causes mixing between the even and odd skyrmion sectors which lifts the ground-state degeneracy. Here we propose the renormalization group flow diagram [Fig. 1] of equation (6). Along the vertical axis at h =0there is a continuous (Landau-forbidden), or weakly first order, phase transition between the small g Néel state and the large g two-fold degenerate nematic PM phase. This is supported by Fig. 4 of Ref. 3. Along the axis where the absolute value of h is large the NATURE PHYSICS www.nature.com/naturephysics 5

transition from the anisotropic Néel state (where there is Néel long-range order but S i S i+ˆx = S i S i+ŷ ) into the PM state [this transition happens at the lower boundary of the diagonal strip in Fig. discussed below] is in the usual O(3) universality class. In the large g region of Fig. 1, where the system remains PM, tuning h from negative to positive encounters a first order phase transition at h =0while maintaining a nonzero S =1gap. Returning to the numerical results of Ref. 3, note that Jiang et al. did study the effect of explicit rotation symmetry breaking on their results by introducing anisotropy in the NN exchange constants in the x and y directions, so that J 1y >J 1x. They found for 0 h (J 1y J 1x )/J 1y 1 there is always (for some range of J /J 1y ) an intermediate PM phase between the Néel and stripe ordered phases [see Fig. for a schematic illustration]. Within this PM phase, they found no evidence of a phase transition, suggesting that it is all one phase. Remarkably it is found that not only does this PM phase survive for 0.55 J /J 1 0.555 in the isotropic limit h 0, but it also includes the case J =0and h =1, where the system consists of a decoupled array of spin-1 chains. As it is independently known that the spin-1 AF chain is in the same phase as the spin-1 AKLT chain, this observation nicely connects the results of the J 1 J model to those obtained 6 6 NATURE PHYSICS www.nature.com/naturephysics

SUPPLEMENTARY INFORMATION for H K. 0.6 0.5 stripe order 0.4 J /J 1y 0.3 0. Neel order 0.1 paramagnetic (a) region 0.0 0.0 0. 0.4 0.6 0.8 1.0 (J J )/J 1y 1x 1y Finally, the other interesting scenario 4, namely, the gapless state of Eq. (6) in main text at Θ=π can be viewed as the critical state between two different SPTs (one at Θ=0and the other Θ=π). Presumably such critical state can be obtained by proliferating the domain walls in the nematic order parameter in our nematic PM phase. Further works are definitely deserved. 3 Exact diagonalization results for the the S =1J 1 -J model Exact diagonalization was performed on 4 4 square lattices with periodic boundary conditions for the spin-1 J 1 -J antiferromagnetic Heisenberg model H = J 1 S i S j + J S j S k. (7) ij jk The spin singlet and spin triplet gaps at momenta q =(π, 0) and (π, π) are presented for different values of J /J 1 in Fig. 3. The global spin-1 gap is given by the minimum of the blue and dashed black curves. As a result it exhibits a kink consistent with that reported in Ref. 3. What is notewor- NATURE PHYSICS www.nature.com/naturephysics 7

thy are (1) the spin-0 gap plunges in the range of J /J 1 where the nematic quantum PM state is expected. This presumably reflects the small splitting (due to quantum tunneling) between the two states that would be degenerate in the thermdynamic limit. () As the (π, 0) triplet gap vanishes as J /J 1 approaches the PM to stripe phase boundary, the (π, π) triplet gap steadily increases. The reverse is true as J /J 1 approaches the PM to Neel phase boundary. That within the PM regime, the S=1 (π, π) is small compared to S=1 (π, 0) on the small J /J 1 and large on the large J /J 1 side suggests the existence of two closeby quantum phase transitions plays a key role in the physics. The ground state fidelity susceptibility 5 is presented in Fig. 4. This quantity displays a clear peak providing strong evidence of a quantum phase transition(s) within 0.5 <J /J 1 < 0.6, even on such a small lattice. This result could be interpreted as indicating a single strongly first-order transition (which is inconsistent with the result of Ref. 3). Alternatively it can be taken as evidence of the existence of two (continuous) transitions (favored by the result of Ref. 3). 1 0.8 S=0 (0,0) S=1 (π,0) S=1 (π,π) (α)/j 1 0.6 0.4 0. 4x4 spin-1 J 1 -J model 0 0 0. 0.4 0.6 0.8 1 α=j /J 1 8 NATURE PHYSICS www.nature.com/naturephysics

SUPPLEMENTARY INFORMATION 1000 [1-F 0 (α,α+δα)]/δα 100 10 1 4x4 spin-1 J 1 -J model δα=0.01 0.1 0 0. 0.4 0.6 0.8 1 α=j /J 1 4 Exact diagonalization results for a model interpolating between the parent Hamiltonian and J 1 -J model We consider a model which interpolates between the parent Hamiltonian H K in Eq. () of the main text and the J 1 -J Heisenberg model, H λ = λ 15 J 1 4 K H K + (1 λ) (J 1 S i S j + J S i S j ) ij ij = J 1 S i S j +[J + λ (J 1 / J )] S i S j + λ (higher order terms), ij ij (8) where the higher order terms contain those terms involving 4 or 6 spin operators. This model becomes the parent Hamiltonian 15 J 1 4 K H K at λ =1, and the J 1 -J Heisenberg model at λ =0. We study the behavior of this model for three different J /J 1 values, J /J 1 =0: the J 1 -J Heisenberg model in this case should exhibit Néel order. The results for the interpolating model are shown in Fig. 5 and Fig. 6. In particular the ground state NATURE PHYSICS www.nature.com/naturephysics 9

fidelity susceptibility shown in Fig. 6 has a prominent peak at around λ =0.9, suggesting a phase transition from nematic paramagnet at λ =1to Néel order at λ =0. 7 6 5 S=0 S=1 (λ)/j 1 4 3 1 0 4x4, (15J 1 /4K)λH K +(1-λ)H J =0 0 0. 0.4 λ 0.6 0.8 1 5 0 4x4, (15J 1 /4K)λH K +(1-λ)H J =0, δλ=0.01 [1-F 0 (λ,λ+δλ)]/δλ 15 10 5 0 0 0. 0.4 λ 0.6 0.8 1 J /J 1 =0.54: according to the DMRG result in Ref. 3, the J 1 -J Heisenberg model in this case will be in PM phase. The results for the interpolating model are shown in Fig. 7 and Fig. 8. The ground state fidelity susceptibility shown in Fig. 8 has no peak, suggesting that the nonmagnetic phase at λ =0(J /J 1 =0.54) is also a nematic paramagnet as the ground states of λ =1parent Hamiltonian. 10 NATURE PHYSICS www.nature.com/naturephysics

SUPPLEMENTARY INFORMATION (λ)/j 1 1.8 1.6 1.4 1. 1 0.8 0.6 S=0 S=1 4x4, (15J 1 /4K)λH K +(1-λ)H J =0.54 0.4 0. 0 0 0. 0.4 λ 0.6 0.8 1 1.8 [1-F 0 (λ,λ+δλ)]/δλ 1.6 1.4 1. 1 0.8 0.6 0.4 0. 0 4x4, (15J 1 /4K)λH K +(1-λ)H J =0.54, δλ=0.01 0 0. 0.4 λ 0.6 0.8 1 J /J 1 =1: for this parameter choice the J 1 -J Heisenberg model should be deep inside the stripe magnetic ordered phase. The results for the interpolating model are shown in Fig. 9 and Fig. 10. The ground state fidelity susceptibility shown in Fig. 10 has a prominent peak at around λ =0.8, suggesting a phase transition from nematic paramagnet at λ =1to stripe order at λ =0. NATURE PHYSICS www.nature.com/naturephysics 11

.5 S=0 S=1 (λ)/j 1 1.5 1 4x4, (15J 1 /4K)λH K +(1-λ)H J =1 0.5 0 0 0. 0.4 λ 0.6 0.8 1 18 [1-F 0 (λ,λ+δλ)]/δλ 16 14 1 10 8 6 4 0 4x4, (15J 1 /4K)λH K +(1-λ)H J =1, δλ=0.01 0 0. 0.4 λ 0.6 0.8 1 1. Tanaka, A. & Hu, X. Many-body spin Berry phases emerging from the π-flux state: Competition between antiferromagnetism and the valence-bond-solid state. Phys. Rev. Lett. 95, 03640 (005). URL http://link.aps.org/doi/10.1103/physrevlett.95.03640.. Senthil, T. & Fisher, M. P. A. Competing orders, nonlinear sigma models, and topological terms in quantum magnets. Phys. Rev. B 74, 064405 (006). URL http://link.aps. org/doi/10.1103/physrevb.74.064405. 1 NATURE PHYSICS www.nature.com/naturephysics

SUPPLEMENTARY INFORMATION 3. Jiang, H. C. et al. Phase diagram of the frustrated spatially-anisotropic s =1antiferromagnet on a square lattice. Phys. Rev. B 79, 174409 (009). URL http://link.aps.org/doi/ 10.1103/PhysRevB.79.174409. 4. Xu, C. & Ludwig, A. W. W. Nonperturbative effects of a topological theta term on principal chiral nonlinear sigma models in +1dimensions. Phys. Rev. Lett. 110, 00405 (013). URL http://link.aps.org/doi/10.1103/physrevlett.110.00405. 5. Gu, S.-J. Fidelity approach to quantum phase transitions. Int. J. Mod. Phys. B 4, 4371 4458 (010). URL http://www.worldscientific.com/doi/abs/10.114/ S01797910056335. NATURE PHYSICS www.nature.com/naturephysics 13

Figure 1 The renormalization group flow diagram of equation (6). The fixed points value of g in this figure are all O(1). The proposed renormalization group flow diagram of equation (6). Along the h =0axis the red critical point separates the small g Néel ordered state from the large g nematic paramagnetic state. Such transition if continuous will be an example of Landau-forbidden transition. The blue critical point separates the anisotropic Néel state (a state with Néel long range order but S i S i+ˆx = S i S i+ŷ ) and the anisotropic paramagnetic phase. Such phase transitions should belong to the O(3) universality class. Figure The phase diagram of the J 1 -J model, Eq. 1 of the main text. A schematic reproduction of the phase diagram reported in Ref. 3. The line of J 1y J 1x =0shows the phase diagram for the J 1 -J model, Eq. 1 of the main text, with fourfold rotation symmetry. Figure 3 The global singlet( S=0 ) gap and the triplet( S=1 ) gaps at momenta q =(π, 0) and (π, π) for the spin-1 J 1 -J model on 4 4 lattice obtained by exact diagonalization. The global triplet gap result is the minimum of the blue and black dashed curves. It exhibit a sharp kink consistent with the DMRG results of Ref. 3. Due to the quantum tunneling between the two degenerate nematic PM states on finite lattices, one expects an unique singlet ground state and a small gap for the singlet excitations. The unique ground state for a given α = J /J 1 is a spin singlet and has lattice momentum (0, 0). The singlet gap is small when J /J 1 falls in the region where the nematic PM state exists (it nearly vanishes around α =0.6where S=0 (α =0.6)=0.0033J 1 ). Moreover the lowest energy singlet 14 NATURE PHYSICS www.nature.com/naturephysics

SUPPLEMENTARY INFORMATION excited state has lattice momentum (0, 0), suggesting no translation symmetry breaking in the tentative nematic quantum PM state. The triplet gap has a cusp around α =0.54. The lowest energy S =1states for α < 0.54 have lattice momentum (π, π) consistent with Néel order, and the lowest energy S =1states for α>0.54 have lattice momentum (π, 0) or (0,π) consistent with stripe antiferromagnetic order. Figure 4 Ground state fidelity susceptibility for the spin-1 J 1 -J model on 4 4 lattice. The ground state fidelity F 0 (α, α+δα) = ψ 0 (α+δα) ψ 0 (α) is the overlap between ground states (ψ 0 ) at parameter α and α + δα. There is a sharp peak at around α =0.56. Figure 5 Singlet( S=0 ) and triplet( S=1 ) gaps for the spin-1 interpolating model equation (8) with J =0on 4 4 lattice. Figure 6 Ground state fidelity susceptibility for the spin-1 interpolating model equation (8) with J =0on 4 4 lattice. There is a peak at around λ =0.9, suggesting that this marks the transition point of a nematic paramagnetic phase for λ>0.9 to a Néel ordered state for λ<0.9. Figure 7 Singlet( S=0 ) and triplet( S=1 ) gaps for the spin-1 interpolating model equation (8) with J /J 1 =0.54 on 4 4 lattice. The singlet gap remains small, suggesting that the nematic paramagnet phase persists to J 1 -J model limit (λ =0). NATURE PHYSICS www.nature.com/naturephysics 15

Figure 8 Ground state fidelity susceptibility for the spin-1 interpolating model equation (8) with J /J 1 =0.54 on 4 4 lattice. There is no peak, suggesting no phase transition from nematic paramagnet at λ =1to the nonmagnetic phase at λ =0. Figure 9 Singlet( S=0 ) and triplet( S=1 ) gaps for the spin-1 interpolating model equation (8) with J /J 1 = 1 on 4 4 lattice. The singlet gap remains small for λ > 0.8, suggesting that the nematic paramagnet phase persists in this region. Figure 10 Ground state fidelity susceptibility for the spin-1 interpolating model equation (8) with J /J 1 =1on 4 4 lattice. There is a peak at around λ =0.8, that this marks the transition point of a nematic paramagnetic phase for λ>0.8 to a stripe ordered state for λ<0.8. 16 NATURE PHYSICS www.nature.com/naturephysics