c 2005 Society for Industrial and Applied Mathematics

Transcription

1 SIAM J. MATRIX ANAL. APPL. Vol. XX, No. X, pp. XX XX c 005 Society for Industrial and Applied Mathematics DISTRIBUTIONS OF THE EXTREME EIGENVALUES OF THE COMPLEX JACOBI RANDOM MATRIX ENSEMBLE PLAMEN KOEV AND IOANA DUMITRIU Abstract. We present explicit formulas for the distributions of the extreme eigenvalues of the complex Jacobi ensemble in terms of the hypergeometric function of a matrix argument. Our methods extend to the quaternion Jacobi ensemble case. Key words. random matrix, Jacobi distribution, MANOVA, multivariate beta, hypergeometric function of a matrix argument, eigenvalue AMS subject classifications. 5A5, 60E05, 6H0, 65F5 DOI. XX.XXXX/SXXXXXXXXXXXXXXXX. Introduction. Complex random covariance matrices arise naturally in applications involving wireless communications [, 5, 8]. Various statistical methods, such as hypothesis testing, principal component analysis, canonical correlation analysis, multivariate analysis of variance, etc., are then based on the distributions of the extreme (generalized) eigenvalues of these matrices [3]. In this paper we present explicit formulas for the distributions of the extreme eigenvalues of the complex Jacobi ensemble in terms of the hypergeometric function of a matrix argument. The formulas are natural analogues of the formulas in the real case. Recently, Ratnarajah, Vaillancourt, and Alvo produced formulas for the distributions of the extreme eigenvalues of the complex Wishart ensemble [4]. Their results, too, are the natural analogues of the real case discussed in Muirhead [3]. Such explicit formulas become increasingly important now that there are algorithms for their efficient computation in practice [9, 0]. Our methods easily extend to the less well-known quaternion Jacobi distribution. This paper is organized as follows. In section we introduce the formal definitions of the Wishart and Jacobi ensembles and the hypergeometric function of a matrix argument that we need later on. In section 3 we survey the existing formulas for the distributions of the extreme eigenvalues of the real Jacobi ensemble. We present our main result formulas for the distributions of the extreme eigenvalues of the complex Jacobi ensemble in section 5. In section 6, we introduce the quaternion Jacobi ensemble, and present formulas for the distributions of its extreme eigenvalues. Finally, we present numerical experiments in section 7.. Background. In this section we recall the definitions of the Wishart and Jacobi random matrix ensembles, and of the hypergeometric function of a matrix argument. Received by the editors October, 005; accepted for publication (in revised form) by EDITOR DATE; published electronically DATE. This work was supported by National Science Foundation grant DMS-03486, and by the Miller Institute for Research in Basic Sciences, U.C. Berkeley, CA. Department of Mathematics, Massachusetts Institute of Technology, Cambridge, MA 039 (plamen@math.mit.edu). Department of Mathematics, University of California, Berkeley, CA 9470 (dumitriu@math.berkeley.edu).

2 PLAMEN KOEV and IOANA DUMITRIU We begin with the real and complex Wishart distributions, since they are the building blocks for the respective Jacobi distributions. If the n m real Gaussian random matrix Z is distributed as N(0, I n Σ), then the matrix A Z T Z is called m m real central Wishart matrix with n degrees of freedom and covariance matrix Σ [3, Def. 3..3, p. 8]. Similarly, if the n m complex Gaussian random matrix Z is distributed as N(0, I n Σ), then the matrix A Z H Z is called m m complex central Wishart matrix with n degrees of freedom and covariance matrix Σ [4]. The real and complex central Wishart ensembles are denoted as W m (n, Σ) and CW m (n, Σ), respectively. If A W m (n, Σ) and B W m (n, Σ), then C A(A + B) is said to have (real) Jacobi distribution. Analogously, if A CW m (n, Σ) and B CW m (n, Σ), then C A(A + B) is said to have complex Jacobi distribution. The Jacobi distribution is sometimes called multivariate beta distribution [3] and is closely related to the MANOVA (Multivariate ANalysis Of VAriance) distribution [3] (the difference is that MANOVA studies the matrix AB instead of A(A + B) ). Definition. (Hypergeometric function of a matrix argument). Let p 0 and q 0 be integers, and let X be an m m complex symmetric matrix with eigenvalues x, x,..., x m. The hypergeometric function of a matrix argument X and parameter α > 0 is defined as (.) pf q (α) (a,..., a p ; b,..., b q ; X) where k means is a partition of k, (.) (a) (α) (i,j) k0 k (a ) (α) (a p ) (α) k!(b ) (α) ( a i ) α + j (b q ) (α) C (α) (X), is the generalized Pochhammer symbol, and C (α) (X) is the Jack function [7]. The Jack function C (α) (X) is a symmetric, homogeneous polynomial in the eigenvalues x, x,..., x m of X. We refer the reader to the excellent survey of Stanley [5] for the explicit definition of C (α) (X) as we do not need it here. In the complex case, α, C (α) (X) equals the well-known Schur function (appropriately scaled), in the real case, α, the zonal polynomial, and in the quaternion case, α, the quaternion zonal polynomial ([]). In this paper we are primarily concerned with the complex case, α ; in Section 6 we deal with the case α. We also utilize the multivariate Gamma function of parameter α, defined as m ( Γ m (α) (c) π m(m ) α Γ c i ) for R(c) > m α α. i 3. Survey of the real case. Let A W m (n, Σ), B W m (n, Σ), where n m and n m. Then the matrix C A(A + B) has the Jacobi real distribution. For the distribution of its largest eigenvalue, we have from [3, eq. (6)] (3.) P (λ max (C) < x) Γ() m ( n+n )Γ () m ( m+ Γ () m ( n+m+ )Γ () ) m ( n ) x mn F () ( n, n+m+ ; n+m+ ; xi).

3 EIGENVALUES OF COMPLEX JACOBI ENSEMBLE 3 The distribution of the smallest eigenvalue is closely related to the distribution of the largest one: (3.) P (λ min (C) < x) P (λ min (C) > x) P (A(A + B) > xi) P (A > x(a + B)) P (B(A + B) < ( x)i). The latest expression is now easily evaluated using (3.) to obtain (3.3) P (λ min (C) < x) Γ() m ( n+n )Γ () m ( m+ ) mn Γ () m ( n+m+ x) )Γ () m ( n )( F () ( n, m n+ ; n+m+ ; ( x)i). Interestingly, we have an explicit formula for the density of the smallest eigenvalue of C [, eq. (9)] dens(λ min (C)) mn Γ() m ( n+n )Γ () m ( m+ Γ () m ( n+m+ ) ) x n m ( x) mn )Γ () m ( n F () ( n, m n+ ; n+m+ ; ( x)i m ). 4. Integral formulas for F () and F () functions. Theorem 7.4. in Muir- functions (the real case). head [3] gives integral representations of the F () and F () Here we present their complex versions. Theorem 4.. The F () function has the integral representation (4.) F () (a; c; X) m (c) m (a) m (c a) e tr (XY ) (det Y ) a m det(i Y ) c a m (dy ), valid for R(a) > m, R(c) > m, R(c a) > m. The F () function has the integral representation (4.) F () (a, b; c; X) m (c) m (a) m (c a) det(i XY ) b (det Y ) a m det(i Y ) c a m (dy ), valid for X <, R(a) > m, R(c a) > m. The expression B < C, where B and C are matrices, means that the matrix C B is Hermitian positive definite. Proof. We proceed analogously to the proof of Theorem 7.4. in Muirhead [3, p. 64]. From Proposition.5 in [4] (4.3) m (det Y ) a m det(i m Y ) b m C () (XY )(dy ) when R(a) > m and R(b) > m. (a)() (a + b) () Γ() m (a) m (b) m (a + b) C () (X),

4 4 PLAMEN KOEV and IOANA DUMITRIU To prove (4.) we expand e tr (XY ) k0 k C () (XY ) k! [6, eq. (89), p. 488] and integrate term-by-term using (4.3). To prove (4.) we expand det(i XY ) b (b) () C () (XY ) k! k0 k [6, eq. (90), p. 488] and integrate the integral in (4.) term-by-term using (4.3). 5. The extreme eigenvalues of the complex Jacobi ensemble. Let the matrix C A(A+B) have complex Jacobi distribution, where A CW m (n, Σ), n m and B CW m (n, Σ), n m. We now derive formulas for the distributions of λ max (C) and λ min (C). Theorem 5.. The distribution of λ max (C) is (5.) P (λ max (C) < x) Γ() m (n + n ) m (m) m (n + m) m (n ) xmn F () (n, m n ; n + m; xi). Proof. The density of A is [6, section 8] The joint density of A and B is (det Σ) n m (n ) (det A)n m e tr ( AΣ ) (da). (det Σ) (n+n) m (n ) m (n ) (det A)n m (det B) n m e tr ( (A+B)Σ ) (da)(db). Let U L AL T, where L is the Cholesky factor of A + B. The matrices U and C thus have the same eigenvalues. By repeating, verbatim, the argument of Theorem 3.3. in Muirhead [3, p. 09], we conclude that the density function of U is (5.) m (n + n ) m (n ) m (n ) (det U)n m det(i U) n m. To compute the distribution of the largest eigenvalue of U we integrate the density (5.) from 0 to xi (5.3) P (U < xi) Γ() m (n + n ) (det U) n m det(i U) n m (du). m (n ) m (n ) 0<U<xI We make an m -dimensional change of variables xy U in (5.3) to obtain (5.4) P (U < xi) Γ() m (n + n )x mn (det Y ) n m det(i xy ) n m (dy ). m (n ) m (n ) We use (4.) to evaluate the integral in (5.4) and obtain (5.). Corollary 5.. The distribution of λ min (C) is (5.5) P (λ min (C) < x) Γ() m (n + n ) m (m) m (n + m) m (n ) ( x)mn F () (n, m n ; n + m; ( x)i). Proof. Directly from (3.) and (5.).

5 EIGENVALUES OF COMPLEX JACOBI ENSEMBLE 5 6. The quaternion case. The quaternion Wishart and Jacobi ensembles are somewhat less known in the world of multivariate statistics and its applications than the real and complex ones, just as the Gaussian Symplectic ensemble is somewhat less known in the quantum physics and mechanical statistics world than the Orthogonal and Unitary ones. One reason for this discrimination is the fact that quaternion normal variables are harder to sample from in existing software. The triad real-complex-quaternion (or equivalently, using the orthogonal symmetry invariance, orthogonal-unitary-symplectic) is what defines the threefold way in random matrix theory (see Dyson, [4]): the three division algebras over the real numbers, over which the concept of a multivariate normal variable is well-defined, and random matrix models of i.i.d. standard Gaussians can be constructed. A quaternion standard Gaussian variable is defined as q (n +n i+n 3 j+n 4 k), where n, n, n 3, n 4 are standard real normals, and i, j, k are the quaternion roots of, i.e., i j k ijk. The quaternion conjugate for q a + a i + a 3 j + a 4 k is q a a i a 3 j a 4 k; qq q a + a + a 3 + a 4. If the n m quaternion Gaussian matrix Y is distributed as HN(0, I n Σ), the matrix A Y D Y has the quaternion central Wishart distribution with n degrees of freedom and covariance matrix Σ. Here Y D denotes the dual (or quaternion conjugate transpose) matrix of Y ; if Y D Y, the matrix Y is called self-dual (and has real eigenvalues). We denote by HW m (n, Σ) the quaternion central Wishart ensemble. Similarly, if A HW m (n, Σ) and B HW m (n, Σ), the matrix C A(A+B) has quaternion Jacobi distribution. Our methods of calculating the extreme eigenvalue distributions for the complex Jacobi ensembles extend immediately to the quaternion case; we hence state the results without repeating the proofs. Theorem 6.. The F ( ) function has the integral representation (6.) F ( ) (a; c; X) Γ ( ) m (c) Γ ( ) m (a)γ ( ) m (c a) e tr (XY ) (det Y ) a m+ det(i Y ) c a m+ (dy ), valid for R(a) > (m ), R(c) > (m ), R(c a) > (m ). The F ( ) function has the integral representation (6.) F ( ) (a, b; c; X) Γ ( ) m (c) Γ ( ) m (a)γ ( ) m (c a) det(i XY ) b (det Y ) a m+ det(i Y ) c a m+ (dy ), valid for X <, R(a) > (m ), R(c a) > (m ). The expression B < C, where B and C are matrices, means that the matrix C B is self-dual positive definite. Theorem 6.. Let C have quaternion Jacobi distribution, C A(A + B), where A HW m (n, Σ) and B HW m (n, Σ). Then the distributions of λ max (C) The notation H stands for William Hamilton, who introduced the quaternions in the 850s.

6 6 PLAMEN KOEV and IOANA DUMITRIU Complex Jacobi, (n,n,m)(3,5,) Complex Jacobi, (n,n,m)(8,0,7) Fig. 7.. The distribution of the largest eigenvalue of the complex Jacobi ensemble. The solid line is the empirical distribution resulting from a Monte Carlo experiment with 0, 000 replications. The circle represents the formula (5.). and λ min (C) are (6.3) (6.4) P (λ max (C) < x) Γ( ) m (n + n )Γ ( ) m (m ) x mn Γ ( ) m (n + m )Γ ( ) m (n ) F ( ) (n, m n ; n + m ; xi); P (λ min (C) < x) Γ( ) m (n + n )Γ ( ) m (m ) ( x) mn Γ ( ) m (n + m )Γ ( ) m (n ) F ( ) (n, m n ; n + m ; ( x)i). 7. Numerical Experiments. We performed extensive numerical tests to confirm the correctness of Theorems 5. and 6. against Monte Carlo experiments; we used MATLAB [] to compute and plot our distributions. We detail several of those tests here whose results were typical. In the complex case we choose two sets of parameters (n, n, m) and for each set compared the empirical distribution from 0, 000 replications with the analytical formula (5.), computed using the algorithms from [9, 0]. In both cases, the results were a perfect visual match see Figure 7.. We generated our matrices in MATLAB as CA/(A+B), where A CW m (n, I), B CW m (n, I). In turn A was generated as AZ *Z/, where Zrandn(n, m)+i*randn(n, m), and B was generated analogously. In the quaternion case, we side-stepped the need to simulate quaternion variables. Instead, we tested with the very elegant (real, tridiagonal) β-jacobi matrix constructed by Sutton [6, Chapter 5]. For β 4 it has the same eigenvalue distribution as the quaternion Jacobi matrix C A(A + B), A HW m (n, Σ), B HW m (n, Σ). Note that we can assume that Σ I without any loss of generality [3, Thm ].

7 EIGENVALUES OF COMPLEX JACOBI ENSEMBLE 7 Quaternion Jacobi, (n,n,m)(3,5,) Quaternion Jacobi, (n,n,m)(8,0,7) Fig. 7.. The distribution of the largest eigenvalue of the quaternion Jacobi ensemble. The solid line is the empirical distribution resulting from a Monte Carlo experiment with 0, 000 replications. The circle represents the formula (6.3). The β-jacobi matrix is defined as Q Z T Z, where c n s n c n Z c n s... n... s c, with c s c k Beta ( β (n m + k), β (n m + k) ), s k c k ; c k Beta ( β k, β (n + n m + k + ) ), s k c k. We plotted the empirical distribution of the eigenvalues of Q against the analytical formula (6.3) in Figure 7., for the same triplets (n, n, m) we used in the complex Jacobi case. Once again we obtained a perfect visual match. REFERENCES [] P.-A. Absil, A. Edelman, and P. Koev, On the largest principal angle between random subspaces. Linear Algebra Appl., to appear. [] P. Chen, G. Genello, and M. Wicks, Estimating the number of signals in presence of colored noise, in Radar Conference 004. Proceedings of the IEEE, 6 9 April 004, pp [3] A. G. Constantine, Some non-central distribution problems in multivariate analysis, Ann. Math. Statist., 34 (963), pp [4] F. J. Dyson, The threefold way. Algebraic structures of symmetry groups and ensembles in Quantum Mechanics, J. Math. Phys., 3 (963), pp [5] H. Gao, P. Smith, and M. Clark, Theoretical reliability of MMSE linear diversity combining in Rayleigh-fading additive interference channels, IEEE Transactions on Communications, 46 (998), pp [6] A. T. James, Distributions of matrix variates and latent roots derived from normal samples, Ann. Math. Statist., 35 (964), pp [7] J. Kaneko, Selberg integrals and hypergeometric functions associated with Jack polynomials, SIAM J. Math. Anal., 4 (993), pp [8] M. Kang and M.-S. Alouini, Largest eigenvalue of complex Wishart matrices and performance analysis of MIMO MRC systems, IEEE Journal on Selected Areas in Communications, (003), pp

8 8 PLAMEN KOEV and IOANA DUMITRIU [9] P. Koev, Software for computing the hypergeometric function of a matrix argument. Available from [0] P. Koev and A. Edelman, The efficient evaluation of the hypergeometric function of a matrix argument. Math. Comp., to appear. [] I. G. Macdonald, Symmetric functions and Hall polynomials, Oxford University Press, New York, Second ed., 995. [] The MathWorks, Inc., MATLAB Reference Guide, Natick, MA, 99. [3] R. J. Muirhead, Aspects of multivariate statistical theory, John Wiley & Sons Inc., New York, 98. [4] T. Ratnarajah, R. Vaillancourt, and M. Alvo, Eigenvalues and condition numbers of complex random matrices, SIAM J. Matrix Anal. Appl., 6 (004/05), pp [5] R. Stanley, Some combinatorial properties of Jack symmetric functions, Adv. Math., 77 (989), pp [6] B. Sutton, The stochastic operator approach to random matrix theory, PhD thesis, Massachusetts Institute of Technology, 005.