MATRIX KERNELS FOR THE GAUSSIAN ORTHOGONAL AND SYMPLECTIC ENSEMBLES

Transcription

1 Ann. Inst. Fourier, Grenoble 55, 6 (5), MATRIX KERNELS FOR THE GAUSSIAN ORTHOGONAL AND SYMPLECTIC ENSEMBLES b Craig A. TRACY & Harold WIDOM I. Introduction. For a large class of finite N determinantal processes the limiting distribution, as N, of the right-most particle is epressible as the Fredholm determinant of an operator K Air with kernel A()A () A ()A() = A( + z) A( + z) dz, where A() = Ai() is the Air function. (See [5], [11] for recent reviews.) Tpicall these results are proved b first establishing that for finite N the distribution of the right-most particle is the Fredholm determinant of an operator K N. The limit theorem then follows once one proves K N K Air in trace norm. The classic eample is the finite N Gaussian Unitar Ensemble (GUE) where K N is the Hermite kernel [6] and the limit N is the edge scaling limit of the largest eigenvalue. In this case the unitar invariance of the underling probabilit measure is manifest. As is well known one can also consider the edge scaling of the largest eigenvalue for the Gaussian Orthogonal Ensemble (GOE) or the Gaussian Smplectic Ensemble (GSE) [9]. These limiting distributions are now widel believed to describe the universal behavior of a right-most particle for a large class of processes under additional smmetr restrictions [3], [8]. In Kewords: random matrices, Gaussian orthogonal, smplectic ensembles. Math. classification: 6F99, 47B34.

2 198 Craig A. TRACY & Harold WIDOM the orthogonal and smplectic ensembles, the distribution of the largest eigenvalue is the square root of a Fredholm determinant of an operator K N whose kernel is now a matri kernel (of course, different K N for different ensembles). In [9] the derivation of the limiting distributions for the largest eigenvalue did not use limiting kernels, as it did for the ensembles scaled in the bulk. Instead, differential equations determining the the finite N distributions were obtained first and then a limiting argument was applied to these. The edge-scaled kernels are alread in the literature [], [1], but as far as we know limit theorems in the operator norms that give convergence of the determinants are not. Because of the current interest in these ensembles, it seems useful to state and prove such theorems for the matri kernels. For GSE we shall establish convergence in trace norm to a limiting operator K GSE. The GOE case is more awkward and for that we shall use weighted Hilbert spaces and an etension of the determinant involving the regularized -determinant, and convergence to K GOE will be in a combination of Hilbert-Schmidt and trace norms. These will give convergence of the determinants. II. The finite N kernels. We shall follow the notation of [1], more or less. For both finite N ensembles we denote b K N (, ) the matri kernel (and K N the corresponding operator) such that the epected value of k (1 + f(λ k)) is given b ( ) (.1) E (1 + f(λ k )) = det (I + K N f). k Here f is a general function and in the determinant it denotes multiplication b f. In particular the probabilit of finding no eigenvalues in a set J is given b (.) det (I KN χ J ). Here are the formulas for K N. We set ε() = 1 sgn, denote b ε the operator with kernel ε( ), and denote b ϕ n the harmonic oscillator wave functions. ANNALES DE L INSTITUT FOURIER

3 MATRIX KERNELS FOR THE GAUSSIAN ORTHOGONAL 199 For GSE, with weight function e and N odd, we define N 1 N S N (, ) = ϕ n () ϕ n ()+ ϕ N 1() εϕ N (), IS N (, ) = n= N 1 n= N 1 S N D(, ) = n= Then (.3) K N (, ) = 1 N εϕ n () ϕ n ()+ εϕ N 1() εϕ N (), ϕ n () ϕ n() N ϕ N 1() ϕ N (). ( ) SN (, ) S N D(, ). IS N (, ) S N (, ) For GOE, with weight function e / and N even we define S N (, ), IS N (, ) and S N D(, ) b eactl the same formulas and then ( ) S (.4) K N (, ) = N (, ) S N D(, ). IS N (, ) ε( ) S N (, ) Despite their apparent similarit the kernels for GSE and GOE have ver different properties. First, the functions εϕ n, which do not vanish at ± when n is even, make it appear that the entries of K N (, ) do not vanish at ±, but the are actuall eponentiall small for GSE. (See derivation in Section 8 of [1]). Thus the f in (.1) can be quite general and even grow at ±, and K N f is a trace class operator on L L. This is not the case for GOE. First, the entries do not all vanish at ± ; second, the discontinuit in ε( ) prevents K N f from being a trace class operator. (The discussion in Section 9 of [1] skirted these issues.) To take care of these problems we use weighted L spaces and a generalization of the determinant. Let ρ be an weight function such that ρ 1 L 1 and such that all ϕ n L (ρ). 1 For an such ρ the matri K N is a Hilbert- Schmidt operator on L (ρ) L (ρ 1 ), and this is the space on which K N acts. The diagonal entries of K N are finite rank, hence trace class. Now the definition of determinant etends to Hilbert-Schmidt operator matrices T with trace class diagonal entries b setting det(i T ) = det (I T ) e tr T, where tr T denotes the sum of the traces of the diagonal entries of T and det is the regularized -determinant. 3 It follows from an identit for - determinants ([4], p. 169) that for this etended definition we still have the 1 Recall that L (ρ) ={h : h() ρ()d < }. That ε is a Hilbert-Schmidt operator from L (ρ) tol (ρ 1 ) is equivalent to ρ 1 L 1. 3 If T is a Hilbert-Schmidt operator with eigenvalues µ k then det (I T ) = (1 µk ) e µ k. See [4], Sec. IV.. TOME 55 (5), FASCICULE 6

4 Craig A. TRACY & Harold WIDOM relation det (I T 1 )(I T ) = det (I T 1 ) det (I T ). Using this fact, and defining det (I K N f) in this wa, 4 the discussion of [1] can be carried through for an f L. III. Scaling GSE at the edge. Here we follow the notation of Section VII of [9] and denote our scaling transformation b τ, so that τ() = N +. N 1/6 We shall consider onl the limit of (.). We denote here b J an measurable set which is bounded below and replace J in (.) b τ(j). We shall compute the matri kernel K GSE such that the limit of the determinant in (.) as N is equal to det (I K GSE χ J ). Observe first that the determinant is unchanged if the kernel K N (, ) χ τ(j) () is replaced b τ K N (τ(), τ()) χ J (), where τ = 1/ N 1/6. It is also unchanged if the upper-right entr is multiplied b τ and the lower-left entr divided b τ. (We shall use the same notation for the modified kernel.) Thus it is a matter of determining the limit, in trace norm, of the entries of the modified matri kernel. and set then If we write ϕ() = ( ) 1/4 N ϕ N (), ψ() = S N (, ) = N 1 n= ( ) 1/4 N ϕ N 1(), ϕ n () ϕ n (), S N (, ) =S N(, )+ψ() εϕ(). 4 The so-defined det (I K N f) is independent of the choice of ρ, since eigenfunctions corresponding to nonzero eigenvalues belong to L (ρ) L (ρ 1 ) for all permissible ρ. In fact it is given b the usual Fredholm epansion, modified for matri kernels. We prefer the operator definition since it will enable us to establish convergence of the determinants more easil. ANNALES DE L INSTITUT FOURIER

5 MATRIX KERNELS FOR THE GAUSSIAN ORTHOGONAL 1 Formula (57) of [9] is in our notation S N(, ) = [ϕ( + z) ψ( + z)+ψ( + z) ϕ( + z)] dz. Define ϕ τ = ϕ τ and ψ τ = ψ τ. Then the substitution z τ z in the integral shows that τ S N (τ(), τ()) = τ [ϕ τ ( + z) ψ τ ( + z)+ψ τ ( + z) ϕ τ ( + z)] dz. It follows from results on the asmptotics of Hermite polnomials that (3.5) lim τ ϕ τ () = lim τ ψ τ () = 1 A() N N pointwise, where A() denotes the Air function Ai(), and that there are estimates (3.6) τ ϕ τ () =O(e ), τ ψ τ () =O(e ) which hold uniforml in N and for bounded below. (There is a better bound but this one is good enough for our purposes. See [7], p. 43.) We shall show that this implies that if then K Air (, ) = A( + z)a( + z)dz, τ S N(τ(), τ()) K Air (, ) in trace norm on an space L (s, ). To show that τ ϕ τ ( + z) ψ τ ( + z) dz 1 K Air(, ), (the other half of τ SN (τ(), τ()) is treated similarl) we write the difference as [τ ϕ τ ( + z) 1/ A( + z)] τ ψ τ ( + z) dz + 1/ A( + z)[τ ψ τ ( + z) 1/ A( + z)] dz. Each summand is an integral over z of rank one kernels, and the trace norm of an integral is at most the integral of the trace norms. Thus the trace norm of the first summand is at most (3.7) τ ϕ τ ( + z) 1/ A( + z) τ ψ τ ( + z) dz, TOME 55 (5), FASCICULE 6

6 Craig A. TRACY & Harold WIDOM where the first L norm is taken with respect to the variable and the second with respect to the variable. It follows from (3.6) that the second norm is O(e z ) uniforml in N, and from (3.6) and (3.5) that the first norm is also O(e z ) uniforml in N and tends pointwise to zero. Thus the integral has limit zero. The same argument applies to the other integral. Thus SN scales to K Air and it remains to consider τ ψ τ ()(εϕ) τ (), the last summand in τ S N (τ(), τ()). Of course τ ψ τ () 1 A() in trace norm. Define We have c ϕ = 1 Φ τ () = (εϕ) τ () =c ϕ ϕ() d, c ψ = 1 τ φ τ (z) dz, Ψ τ () = τ() ϕ(z) dz = ψ() d, τ ψ τ (z) dz. τ ϕ τ (z) dz = Φ(). (Since N isoddsoisϕ, soc ϕ =.) From (3.5) and (3.6) it follows that this converges to 1 A(z) dz in the norm of an space L (s, ). Thus we have shown that τ S N (τ(), τ()) K Air (, ) 1 A() A(z) dz in the trace norm of operators on an space L (s, ). Net, observe that S N D(, ) = S N (, ) and so (recall that we multipl the upper-right corner b τ ) τ SN D(τ(), τ()) = τ S N (τ(), τ()). Since the limiting relation we found above for S N holds even after taking we see that τ SN D(τ(), τ()) K Air (, ) 1 A() A() in trace norm as N. If we recall that we divide the lower-left corner b τ we see that it remains to find the limit of IS N (τ(), τ()) = εs N(τ(), τ()) + εψ(τ()) εϕ(τ()). ANNALES DE L INSTITUT FOURIER

7 MATRIX KERNELS FOR THE GAUSSIAN ORTHOGONAL 3 The last term equals (c ψ Ψ τ ()) Φ τ (). If we make the substitution z τ z in the integral for εsn (τ(), τ()) it becomes (3.8) [(εϕ τ )( + z) τ ψ τ ( + z)+(εψ) τ ( + z) τ ϕ τ ( + z)] dz. Replacing εϕ τ and εψ τ b what the are in terms of Φ τ and Ψ τ gives [Φ τ ( + z) τ ψ τ ( + z)+ψ τ ( + z) τ ϕ τ ( + z)] dz + c ψ Φ τ (). Putting these together we obtain IS N (τ(), τ()) = [Φ τ ( + z) τ ψ τ ( + z)+ψ τ ( + z) τ ϕ τ ( + z)] Now we use (3.5) and (3.6) as before, and deduce that IS N (τ(), τ()) in trace norm. Thus if we set IS(, ) = K A (z,) dz + 1 A(z) dz S(, ) =K Air (, ) 1 A() A(z) dz, SD(, ) = K Air (, ) 1 A() A(), K A (z,) dz + 1 then we have shown that for GSE τ K N (τ(), τ()) K GSE (, ) := 1 A(z) dz dz +Ψ τ ()Φ τ (). A(z) dz, A(z) dz ( S(, ) ) SD(, ) IS(, ) S(,) in trace norm as N. In particular, the probabilit that no eigenvalue lies in τ(j) tends to det (I KGSE χ J ) as N. TOME 55 (5), FASCICULE 6

8 4 Craig A. TRACY & Harold WIDOM IV. Scaling GOE at the edge. Before computing limits let us see what the norm of a rank one kernel u()v() is when thought of as taking a space L (ρ 1 ) to a space L (ρ ). The operator, denoted b u v, takes a function h L (ρ 1 )tou(v, h), and so its norm is the L (ρ ) norm of u times the norm of v in the space dual to L (ρ 1 ), which is L (ρ 1 1 ). Thus (4.9) u v = u L (ρ ) v L (ρ 1 ). 1 We determine the limit of the scaled kernel τ K N (τ(),τ()) where K N is now given b (.4). The space on which this acts is L (ρ) L (ρ 1 ) with an fied permissible ρ which we now assume has at most polnomial growth at +. 5 As before, we multipl the upper-right corner b τ and divide the lower-left corner b τ, but do not change notation. For τ SN (τ(), τ()), the main part of τ S N (τ(), τ()), consider the analogue of (3.7). Since this kernel takes L (ρ) to itself (4.9) gives for the analogue here (4.1) τ ϕ τ ( + z) 1/ A( + z) L (ρ) τ ψ τ ( + z) L (ρ 1 )dz. The second factor is O(e z ) uniforml in N since τ ψ τ ( + z) =O(e z ) and ρ 1 L 1. The first factor is at most the square root of s τ ϕ τ ( + z) 1/ A( + z) ρ()d for some s. Using (3.5) and (3.6) and the fact that ρ has at most polnomial growth we see that this is uniforml O(e z/ ) and converges to zero pointwise. Thus (4.1) has limit zero. The other integral arising in the amptotics of τ S N (τ(), τ()) is similar, and we deduce that τ S N(τ(),τ()) K Air (, ) in trace norm, as an operator on L (ρ). It remains to consider τ ψ τ ()(εϕ) τ (). First, τ ψ τ () 1 A() in the space L (ρ), again b (3.5) and (3.6) and the fact that ρ has at most polnomial growth. Now (εϕ) τ () =c ϕ Φ(), 5 All functions are thought of as defined on some interval (s, ). ANNALES DE L INSTITUT FOURIER

9 MATRIX KERNELS FOR THE GAUSSIAN ORTHOGONAL 5 and since N is even for GOE so is ϕ and c ϕ. In fact c ϕ 1 N and so we see that (εϕ) τ () 1 1 A(z)dz uniforml, and so in the space L (ρ 1 ). Thus, τ S N (τ(),τ()) K Air (, )+ 1 A() (1 ) A(z) dz in trace norm as an operator on L (ρ). Consider now the upper-right corner of the modified matri, τ SN D(τ(), τ()) = τ S N (τ(), τ()) τ ψτ () ϕ τ (), whose limit we want to compute as an operator from L (ρ 1 )tol (ρ). In the application of (4.9) both norms are that of L (ρ). In the verification that τ S N(τ(),τ()) K Air (, ) we obtain an integral of the form (4.1) where the second factor in the integrand is replaced b τ ψ τ ( + z) L (ρ). This is bounded uniforml in N and z because (3.6) holds also even after taking derivatives. The first integrand is estimated as before, so the integral tends to zero in trace norm. We also have τ ψ τ () 1 A() and τ ϕ τ () 1 A() inl (ρ), and so we find that τ SN D(τ(),τ()) K Air (, ) 1 A()A() in trace norm as before, but now as operators from L (ρ 1 )tol (ρ). For the lower-left corner, an operator now from L (ρ) tol (ρ 1 )so in (4.9) both norms are L (ρ 1 ) norms, we have as before IS N (τ(),τ()) = εs N (τ(),τ()) + εψ(τ())εϕ(τ()). The last term equals Ψ τ ()(c ϕ Φ τ ()). Replacing εϕ τ and εψ τ in (3.8) b what the are in terms of Φ τ and Ψ τ now gives [Φ τ ( + z)τ ψ τ ( + z)+ψ τ ( + z)τ ϕ τ ( + z)]dz + c ϕ Ψ τ (), as TOME 55 (5), FASCICULE 6

10 6 Craig A. TRACY & Harold WIDOM so we obtain now the identit IS N (τ(),τ())= [Φ τ (+z)τ ψ τ ( + z)+ψ τ ( + z)τ ϕ τ ( + z)]dz + c ϕ Ψ τ () Ψ τ ()(c ϕ Φ τ ()). Now we deduce that ( IS N (τ(), τ()) K A (z,) dz + 1 A(z) dz + as trace class operators from L (ρ) tol (ρ 1 ). Finall, ε(τ() τ()) = ε( ), so this is unchanged. Thus if we set S(, ) =K Air (, )+ 1 ) (1 A() A(z) dz, A(z) dz ) A(z) dz SD(, ) = K Air (, ) 1 A()A(), IS(, ) = K A (z,)dz + 1 ( ) A(z)dz + A(z)dz A(z)dz, then we have shown that for GOE ( ) τ S(, ) SD(, ) K N (τ(),τ()) K GOE (, ) := IS(, ) ε( ) S(, ) as N in the sense that the kernels converge in the Hilbert-Schmidt norm of operators on L (ρ) L (ρ 1 ) and the diagonals converge in trace norm. (In fact all entries converge in trace norm, ecept for the fied Hilbert-Schmidt operator ε.) In particular, it follows from the continuit of the -determinant in Hilbert-Schmidt norm and the trace in trace norm that the probabilit that no eigenvalue lies in τ(j) tends to det (I KGOE χ J ) as N. Acknowledgments. This work was supported b National Science Foundation under grants DMS (first author) and DMS-4398 (second author). ANNALES DE L INSTITUT FOURIER

11 MATRIX KERNELS FOR THE GAUSSIAN ORTHOGONAL 7 BIBLIOGRAPHY [1] P.L. FERRARI, Polnuclear growth on a flat substrate and edge scaling of GOE eigenvalues, Commun. Math. Phs., 5 (4), [] P.J. FORRESTER, T. NAGAO and G. HONNER, Correlations for the orthogonalunitar and smplectic-unitar transitions at the soft and hard edges, Nucl. Phs., B 553 (1999), [3] J. BAIK and E.M. RAINS, Smmetrized random permutations, in Random Matri Models and Their Applications, eds. P.M. Bleher and A.R. Its, Cambridge Univ. Press, (1), [4] I.C. GOHBERG and M.G. KREIN, Introduction to the Theor of Linear Nonselfadjoint Operators, Transl. Math. Monogr., 35, Providence, RI: Amer. Math. Soc., [5] K. JOHANSSON, Toeplitz determinants, random growth and determinantal processes, in Proc. of the International Congress of Mathematicians, Vol. III, Higher Education Press,, [6] M.L. MEHTA, Random Matrices, London: Academic Press, [7] F.W.J. OLVER, Asmptotics and Special Functions, New York: Academic Press, [8] M. PRÄHOFER and H. SPOHN, Universal distributions for growth processes in 1+1 dimensions and random matrices, Phs. Rev. Letts., 84 (), [9] C.A. TRACY and H. WIDOM, On orthogonal and smplectic matri ensembles, Commun. Math. Phs., 177 (1996) [1] C.A. TRACY and H. WIDOM, Correlation functions, cluster functions and spacing distributions for random matrices, J. Stat. Phs., 9 (1998), [11] C.A. TRACY and H. WIDOM, Distribution functions for largest eigenvalues and their applications, Proc. of the International Congress of Mathematicians, Vol. I, Higher Education Press, (), Craig A. TRACY, Universit of California Department of mathematics Davis, CA (USA) trac@math.ucdavis.edu Harold WIDOM, Universit of California Department of mathematics Santa Cruz, CA 9564 (USA) widom@math.ucsc.edu TOME 55 (5), FASCICULE 6

12