Ranks and determinants of the sum of matrices from unitary orbits

Transcription

1 Linear and Multilinear Algebra, Vol. 56, Nos. 1 2, Jan. Mar. 2008, Ranks and determinants of the sum of matrices from unitary orbits CHI-KWONG LI*y, YIU-TUNG POONz and NUNG-SING SZEx ydepartment of Mathematics, College of William & Mary, Williamsburg, VA 23185, USA zdepartment of Mathematics, Iowa State University, Ames, IA 50011, USA xdepartment of Mathematics, University of Connecticut, Storrs, CT 06269, USA Communicated by C. Tretter (Received 30 January 2007; in final form 27 March 2007) The unitary orbit UðAÞ of an nn complex matrix A is the set consisting of matrices unitarily similar to A. Given two nn complex matrices A and B, ranks and determinants of matrices of the form X þ Y with ðx, Y Þ2UðAÞUðBÞ are studied. In particular, a lower bound and the best upper bound of the set RðA, BÞ ¼frankðX þ YÞ : X2UðAÞ, Y2UðBÞg are determined. It is shown that ða, BÞ ¼fdetðX þ YÞ : X2UðAÞ, Y2UðBÞg has empty interior if and only if the set is a line segment or a point; the algebraic structure of matrix pairs (A, B) with such properties are described. Other properties of the sets R(A, B) and ða, BÞ are obtained. The results generalize those of other authors, and answer some open problems. Extensions of the results to the sum of three or more matrices from given unitary orbits are also considered. Keywords: Rank; Determinant; Matrices; Unitary orbit 2000 Mathematics Subject Classifications: 15A03; 15A15 1. Introduction Let A2M n. The unitary orbit of A is denoted by UðAÞ ¼fUAU : U U ¼ I n g: Evidently, if A is regarded as a linear operator acting on C n, then UðAÞ consists of the matrix representations of the same linear operator under different *Corresponding author. ckli@math.wm.edu Dedicated to Professor Yik-Hoi Au-Yeung on the occasion of his 70th birthday. Linear and Multilinear Algebra ISSN print/issn online ß 2008 Taylor & Francis DOI: /

2 106 C.-K. Li et al. orthonormal bases. Naturally, UðAÞ captures many important features of the operator A. For instance, A is normal if and only if UðAÞ has a diagonal matrix; A is Hermitian (positive semi-definite) if and only if UðAÞ contains a (nonnegative) real diagonal matrix; A is unitary if and only if UðAÞ has a diagonal matrix with unimodular diagonal entries. There are also results on the characterization of diagonal entries and submatrices of matrices in UðAÞ; see [14,20,23,30] and their references. In addition, the unitary orbit of A has a lot of interesting geometrical and algebraic properties, see [11]. Motivated by theory as well as applied problems, there has been a great deal of interest in studying the sum of two matrices from specific unitary orbits. For example, eigenvalues of UAU þ VBV for Hermitian matrices A and B were completely determined in terms of those of A and B (see [16] and its references); the optimal norm estimate of UAU þ VBV was obtained (see [10] and its references); the range of values of detðuau þ VBV Þ for Hermitian matrices A, B2M n was described, see [15]. Later, Marcus and Oliveira [26,29] conjectured that if A, B2M n are normal matrices with eigenvalues a 1,..., a n and b 1,..., b n, respectively, then for any unitary U, V2M n, detðuau þ VBV Þ always lies in the convex hull of ( ) PðA, BÞ ¼ Yn a j þ b ðjþ : is a permutation of f1,..., ng : ð1:1þ j¼1 In connection to this conjecture, researchers [2 5,7,8,12,13] considered the determinantal range of A, B2M n defined by ða, BÞ ¼fdetðA þ UBU Þ: U is unitaryg: This can be viewed as an analog of the generalized numerical range of A and B defined by WðA, BÞ ¼ftrðAUBU Þ: U is unitaryg, which is a useful concept in pure and applied areas; see [18,21,25] and their references. In this article, we study some basic properties of the matrices in UðAÞþUðBÞ ¼fX þ Y: ðx, Y Þ2UðAÞUðBÞg: The focus will be on the rank and the determinant of these matrices. Our article is organized as follows. In section 2, we obtain a lower bound and the best upper bound for the set RðA, BÞ ¼frankðX þ Y Þ: X 2UðAÞ, Y 2UðBÞg; moreover, we characterize those matrix pairs (A, B) such that R(A, B) is a singleton, and show that the set R(A, B) has the form fk, k þ 1,..., ng if A and B have no common eigenvalues. On the contrary if A and B are orthogonal projections, then the rank values of matrices in UðAÞþUðBÞ will either be all even or all odd. In section 3, we characterize matrix pairs ða, BÞ such that ða, BÞ has empty interior, which is

3 Ranks and determinants of the sum of matrices 107 possible only when ða, BÞ is a singleton or a nondegenerate line segment. This extends the results of other researchers who treated the case when A and B are normal [2 4,9]. In particular, our result shows that it is possible to have normal matrices A and B such that ða, BÞ is a subset of a line, which does not pass through the origin. This disproves a conjecture in [9]. In [3], the authors showed that if A, B2M n are normal matrices such that the union of the spectra of A and B consists of 2n distinct elements, then every nonzero sharp point of ða, BÞ is an element in P(A, B). (See the definition of sharp point in section 3.) We showed that every (zero or nonzero) sharp point of ða, BÞ belongs to P(A, B) for arbitrary matrices A, B2M n. In section 4, we consider the sum of three or more matrices from given unitary orbits, and matrix orbits corresponding to other equivalence relations. In the literature, some authors considered the set DðA, BÞ ¼fdetðX YÞ: X 2UðAÞ, Y 2UðBÞg instead of ða, BÞ. Evidently, we have DðA, BÞ ¼ðA, BÞ. It is easy to translate results on D(A, B) to those on ða, BÞ, and vice versa. Indeed, for certain results and proofs, it is more convenient to use the formulation of D(A, B). We will do that in section 3. On the other hand, it is more natural to use the summation formulation to discuss the extension of the results to matrices from three or more unitary orbits. 2. Ranks 2.1. Maximum and minimum rank In [10], the authors obtained optimal norm bounds for matrices in UðAÞþUðBÞ for two given matrices A, B2M n. By the triangle inequality, we have maxfkuau þ VBV k: U, V unitaryg minfka I n kþkb þ I n k: 2 Cg: It was shown in [10] that the inequality is actually an equality. For the rank functions, we have maxfrankðuau þ VBV Þ: U,V unitarygminfrankða I n ÞþrankðB þ I n Þ: 2 Cg: Of course, the right side may be strictly larger than n, and thus equality may not hold in general. It turns out that this obstacle can be overcome easily as shown in the following. THEOREM 2.1 Let A, B2M n and m ¼ minfrankða I n ÞþrankðB þ I n Þ: 2 Cg ¼ minfrankða I n ÞþrankðB þ I n Þ: is an eigenvalue of A Bg: Then maxfrankðuau þ VBV Þ : U, V unitaryg ¼minfm, ng:

4 108 C.-K. Li et al. Proof If is an eigenvalue of A B, then rankða I n ÞþrankðB þ I n Þ2n 1; if is not an eigenvalue of A B, then rankða I n ÞþrankðB þ I n Þ¼2n: As a result, rankða I n ÞþrankðB þ I n Þ will attain its minimum at an eigenvalue of the matrix A B. It is clear that maxfrankðuau þ VBV Þ: U, V unitaryg minfm, ng: It remains to show that there are U, V such that UAU þ VBV has rank equal to minfm, ng. Suppose mn and there is such that rankða I n Þ¼k and rankðb þ I n Þ¼m k. We may replace (A, B) byða I n, B þ I n Þ and assume that ¼ 0. Furthermore, we may assume that km k; otherwise, interchange A and B. Let A ¼ XDY be such that X, Y2M n are unitary, and D ¼ D 1 0 n k with invertible diagonal D 1. Replace A by YAY*, we may assume that A ¼ UD ¼ U 11 U 12 D1 0 U 21 U n k with U ¼ YX. Similarly, we may assume that B ¼ VE ¼ V 11 V 12 0k 0 V 21 V 22 0 E 2 where V is a unitary matrix and E 2 is a diagonal matrix with rank m k. Let W be a unitary matrix such that the first k columns of WU together with the last n k columns of V are linearly independent. That is, if W ¼ W 11 W 12, W 21 W 22, the matrix W 11 U 11 þ W 12 U 21 V 12 W 21 U 11 þ W 22 U 21 V 22 is invertible. If W 11 is invertible, then D 1 W 11 has rank k and so WAW þ B ¼ W 11U 11 þ W 12 U 21 V 12 D1 W 11 D 1 W 21 W 21 U 11 þ W 22 U 21 V 22 0 E 2 has rank m. IfW 11 is not invertible, we will replace W by ~W obtained as follows. By the CS decomposition, there are unitary matrices P 1, Q 1 2M k and P 2, Q 2 2M n k such that ðp 1 P 2 ÞWðQ 1 Q 2 Þ¼ C S I n 2k, S C

5 Ranks and determinants of the sum of matrices 109 q where C ¼ diagðc 1,..., c k Þ with 1 c 1 c k 0 and S ¼ diagð ffiffiffiffiffiffiffiffiffiffiffiffi qffiffiffiffiffiffiffiffiffiffiffiffi 1 c 2 1,..., 1 c 2 k Þ. Then perturb the zero diagonal entries of C slightly to ~C ¼ diagð~c 1,..., ~c k Þ so that ~C q is invertible, and set ~S ¼ diagð ffiffiffiffiffiffiffiffiffiffiffiffi qffiffiffiffiffiffiffiffiffiffiffiffi 1 ~c 2 1,..., 1 ~c 2 k Þ. Then ~W ¼ðP 1 P 2 Þ ~C ~S I n 2k ðq 1 Q 2 Þ ~S ~C will be a slight perturbation of W with invertible ~W 11 ¼ P ~ 1 CQ 1, which can be chosen such that the matrix! ~W 11 U 11 þ ~W 12 U 21 V 12 ~W 21 U 11 þ ~W 22 U 21 V 22 is still invertible. Then ~WA ~W þ B has rank m. Now, assume that rankða I n ÞþrankðB þ I n Þn þ 1 for every 2C. Let a 1,..., a n and b 1,..., b n be the eigenvalues of A and B. We consider two cases. First, suppose a i þ b j 6¼ 0 for some i, j. We may assume that i ¼ j ¼ 1. Applying suitable unitary similarity transforms, we may assume that A and B are unitarily similar to matrices in upper triangular form a 1 0 A 1 and b 1 0 B 1 Since rankða I n ÞþrankðB þ I n Þn þ 1 for every 2C, it follows that rankða 1 I n 1 ÞþrankðB 1 þ I n 1 Þn 1 for every 2C. By induction assumption, there is a unitary V 1 such that detða 1 þ V 1 B 1 V 1 Þ 6¼ 0. Let V ¼½1ŠV 1. Then detða þ VBV Þ¼ða 1 þ b 1 Þ detða 1 þ V 1 B 1 V 1Þ 6¼ 0. Suppose a i þ b j ¼ 0 for all i, j2f1,..., ng. Replacing (A, B) byða a 1 I n, B b 1 I n Þ, we may assume that A and B are nilpotents. If A or B is normal, then it will be the zero matrix. Then rankðaþ þrankðbþ < n, which contradicts our assumption. Suppose neither A nor B is normal and rank Arank B. If n ¼ 3, then rank A ¼ rank B ¼ 2. We may assume that : B C B C A A and B 1 0 0A such that 1, 3, 1, 3 are nonzero. Interchange the last two rows and the last two columns of A to obtain ^A. For U ¼ diagð1, 1, e i Þ, we have e i U ^AU ¼ B A: 0 3 e i 0 Evidently, there is 2½0, 2Þ such that detðu ^AU þ BÞ 6¼ 0.

6 110 C.-K. Li et al. Suppose n 4. Applying suitable unitary similarity transforms, we may assume that both A and B are in upper triangular form with nonzero (1, 2) entries; see [27, Lemma 1]. Modify B by interchanging its first two rows and columns. Then, A and B have the form A 11 A 12 B 11 B 12 and 0 A 22 0 B 22 respectively, so that A 11 ¼ and B 11 ¼ with 6¼ 0, and A 22, B 22 rank Bn þ 2, then are upper triangular nilpotent matrices. If rank A þ rankða 22 I n 2 ÞþrankðB 22 þ I n 2 Þrank A 22 þ rank B 22 rank A þ rank B 4 n 2: If rank A þ rank B ¼ n þ 1, we claim that by choosing a suitable unitary similarity transform, we can further assume that rank A 22 ¼ rank A 1. Then rankða 22 I n 2 ÞþrankðB 22 þ I n 2 Þrank A 22 þ rank B 22 rank A þ rank B 3 ¼ n 2: In both cases, by induction assumption, there is V 2 such that detða 22 þ V 2 B 22 V 2Þ 6¼ 0. Let V ¼ I 2 V 2. Then detða þ VBV Þ¼ detða 22 þ V 2 B 22 V 2Þ 6¼ 0: Now it remains to verify our claim. Suppose A has rank k and rank A þ rank B ¼ n þ 1. Then kðn þ 1Þ=2. Let S be an invertible matrix such that S 1 AS ¼ J is the Jordan form of A. If J has a 22 Jordan block, then we can always permute J so that J ¼ J J 22 with J 11 ¼ and rankðj 22 Þ¼p 1. By QR factorization, write S ¼ U T for some unitary matrix U and invertible upper triangular matrix T ¼ T 11 T 12 : 0 T 22 Then A is unitarily similar to TJT 1 ¼ which has the described property. T! 11J 11 T 1 11, 0 T 22 J 22 T 1 22

7 Ranks and determinants of the sum of matrices 111 Suppose J does not contain any 22 block, then J must have an 11 Jordan block. Otherwise, k ¼ rank A2n=3 and hence rank A þ rank B 2k 4n=3 ¼ n þ n=3 > n þ 1: Now we may assume that J ¼ 0 2 J 12 0 J 22 is strictly upper triangular matrix such that J 12 has only a nonzero entry in the (1, 1)-th position and rank J 22 ¼ k 1. Let ^S be obtained from I n by replacing the (3, 2)-th entry with one, then with ^S 1 J ^S ¼ ^J ¼ ^J 11 J 12 0 J 22 ^J 11 ¼ 0 1 : 0 0 Applying QR factorization on S ^S ¼ U T with unitary U and invertible upper triangular T, then A is unitarily similar to T ^JT 1, which has the described form. g The value m in Theorem 2.1 is easy to determine as we need only to focus on rankða I n ÞþrankðB þ I n Þ for each eigenvalue of A B. In particular, if is an eigenvalue of A, then rankða I n Þ¼n k, where k is the geometric multiplicity of ; otherwise, rankða I n Þ¼n. Similarly, one can determine rankðb þ I n Þ. The situation for normal matrices is even better as shown in the following. COROLLARY 2.2 Suppose A and B are normal matrices such that is the maximum multiplicity of an eigenvalue of A B. Then minfrankða I n Þþ rankðb þ I n Þ: 2Cg equals 2n. Consequently,! maxfrankðuau þ VBV Þ: U, V unitaryg ¼minf2n, ng: Here are some other consequences of Theorem 2.1. COROLLARY 2.3 Let A, B2M n. Then UAU þ VBV is singular for all unitary U, Vif and only if there is 2C such that rankða I n ÞþrankðB þ I n Þ<n. COROLLARY 2.4 Let A2M n, and k ¼ minfrankða I n Þ: is an eigenvalue of Ag: Then maxfrankðuau VAV Þ: U, V unitary g¼minfn,2kg:

8 112 C.-K. Li et al. If k<n=2, then UAU VAV is singular for any unitary U, V2M n. In case A is normal, then n k is the maximum multiplicity of the eigenvalues of A. Partition M n as the disjoint union of unitary orbits. We can define a metric on the set of unitary orbits by dðuðaþ, UðBÞÞ ¼ minfrankðx Y Þ: X 2UðAÞ, Y 2UðBÞg: For example, if A and B are two orthogonal projections of rank p and q, respectively, then dðuðaþ, UðBÞÞ ¼ jp qj; see Proposition 2.8. So, the minimum rank of the sum or difference of matrices from two different unitary orbits has a geometrical meaning. However, it is not so easy to determine the minimum rank for matrices in UðAÞþUðBÞ in general. We have the following observation. PROPOSITION 2.5 Let A, B2M n and 2C be such that rankða I n Þ¼p and rankðb þ I n Þ¼q. Then minfrankðuau þ VBV Þ : U, V unitaryg maxfp, qg: The inequality becomes equality if A I n and B þ I n are positive semi-definite. Proof There exist unitary U, V such that the last n p columns of UðA I n ÞU are zero, and the last n q columns of VðB þ I n ÞV are zero. Then rankðuau þ VBV Þmaxfp, qg. g The upper bound in the above proposition is rather weak. For example, we may have A and B such that maxfrankða I n Þ, rankðb þ I n Þ: 2 Cg ¼n 1 ð2:1þ and rankðuau þ VBV Þ¼1. Example 2.6 Let A ¼ diagð1, 2,..., nþ and B ¼ J A, where J2M n is the matrix having all entries equal to 1/n. Then B has distinct eigenvalues b 1 >>b n such that b 1 >n>b 2 >n 1>b 3 >>b 1 >1. Then (2.1) clearly holds and rankða þ BÞ ¼1. In fact, by Theorem 2.7 subsequently, we know that for any m2f1,..., ng, there are unitary U, V2M n such that rankðuau þ VBV Þ¼m Additional results Here we study other possible rank values of matrices in UðAÞþUðBÞ. The following result shows that if A and B have disjoint spectra, then one can get every possible rank values from the minimum to the maximum value, which is n. THEOREM 2.7 Suppose A, B2M n such that A and B have disjoint spectra, and A þ B has rank k < n. Then for any m 2fk þ 1,..., ng, there is a unitary U such that UAU þ B has rank m. Proof Let A, B2M n satisfy the hypotheses of the theorem. We need only to show that there is a unitary U such that UAU þ B has rank k þ 1. Then we can apply the

9 Ranks and determinants of the sum of matrices 113 argument again to get a unitary ^U such that ^UUAU ^U þ B has rank k þ 2. Repeating this procedure, we will get the desired conclusion. If k ¼ n 1. Then assume VAV* and WBW* are in upper triangular form. For U ¼ W V, we have UAU þ B ¼ W ðvav þ WBW ÞW is invertible. For 1k<n 1. We may assume that A þ B ¼ C ¼ C 11 0 : C 21 0 n k Let A ¼ A 11 A 12 A 21 A 22 and B ¼ B 11 B 12 B 21 B 22 with A 11, B 11 2M k. Note that A 12 6¼ 0. Otherwise, A and B have common eigenvalues since A 22 ¼ B 22. Assume C 21 6¼ 0. We may replace A þ B by VðA þ BÞV for some permutation matrix V2M n of the form V 1 I n k so that the matrix obtained by removing the first row of VðA þ BÞV still has rank k. For notational simplicity, we may assume that V ¼ I n. Since A 12 6¼ 0, we may assume that the first row of A 12 6¼ 0. Otherwise, replace (A, B) by ðvav, VBV Þ for some unitary V ¼ V 1 I n k. Here we still assume that removing the first row of A þ B results in a rank k matrix. Then there exists a small >0 such that for U ¼ diagðe i,1,...,1þ the matrix UAU þ B has rank k þ 1 because removing its first row has rank k, and adding the first row back will increase the rank by 1. Now, suppose C 21 ¼ 0. Then A 21 6¼ 0. Otherwise, A and B have common eigenvalues since A 22 ¼ B 22. Now, C 11 is invertible. Assume that the matrix obtained by removing the first row and first column of C 11 has rank k 1. Otherwise, replace (A, B) by ðvav, VBV Þ by some unitary matrix V of the form V 1 I n k. Since A 12 and A 21 are nonzero, we may further assume that v t ¼½a 1, kþ1,..., a 1n Š 6¼ 0 and u ¼½a kþ1, 1,..., a n1 Š t 6¼ 0. Then there exists a small >0 such that for U ¼ diagðe i,1,...,1þ the matrix UAU þ B has the form ^C 11 ^C 21 ^C12 0 n k!, where ^C 11 is invertible such that removing its first row and first column results in a rank k 1 matrix, only the first row of ^C 12 is nonzero and equal to ðe i 1Þv t, only the first column of ^C 21 is nonzero and equal to ðe i 1Þu. Now, removing the first row and first column of UAU þ B has rank k 1; adding the column ðe i 1Þu (to the left) will increase the rank by 1, and then adding the first row back will increase the rank by 1 more. So, UAU þ B has rank k þ 1. g Note that the assumption that A and B have disjoint spectra is essential. For example, if A, B2M 4 such that A and B are rank 2 orthogonal projections, then UAU þ VBV can only have ranks 0, 2, 4. More generally, we have the following.

10 114 C.-K. Li et al. PROPOSITION 2.8 Suppose A, B2M n such that A and B are orthogonal projections of rank p and q. Then k ¼ rankðuau þ BÞ for a unitary matrix U2M n, if and only if k ¼jp qjþ2j with j0 and kminfp þ q,2n p qg. Proof Suppose UAU ¼ I p 0 n p and VBV ¼ 0 j I q 0 n j q. Then UAU þvbv has rank k ¼jp qjþ2jminfp þ q,2n p qg. Thus, V UAU V þ B has rank k as well. Conversely, consider UAU þ B for a given unitary U. There is a unitary V such that VUAU V ¼ I p 0 n p and VBV ¼ I r 0 s C2 CS I CS S 2 u 0 v, where C and S are invertible diagonal matrices with positive diagonal entries such that C 2 þ S 2 ¼ I t, r þ s þ t ¼ p and r þ t þ u ¼ q. (Evidently, the first r columns of V* span the intersection of the range spaces of UAU* and B, the next s columns of V* span the intersection of the range space of UAU* and the null space of B, the last v columns of V* span the intersection of the null space of UAU* and B, the u columns preceding those span the intersection of the range space of B and the null space of UAU*.) So, UAU þ B has the asserted rank value. g The following result was proved in [24]. THEOREM 2.9 Let A, B2M n. Then UAU þ VBV is invertible for all unitary U, V2M n if and only if there is 2C such that the singular values of A I n and B þ I n lie in two disjoint closed intervals in ½0, 1Þ. Using this result, we can deduce the following. THEOREM 2.10 Let A, B2M n and k2f0,..., ng. Then rankðuau þ VBV Þ¼k for all unitary U, V2M n if and only if one of the following holds. (a) One of the matrices A or B is scalar, and rankða þ BÞ ¼k. (b) k ¼ n and there is 2C such that the singular values of A I n and B þ I n lie in two disjoint closed intervals in ½0, 1Þ. Proof If (a) holds, say, B ¼ I n, then rankðuau þ VBV Þ¼rankðA I n Þ¼kfor all unitary U, V2M n. If (b) holds, then kða I n Þxk>kðB þ I n Þyk for all unit vectors x, y2c n, or kða I n Þxk<kðB þ I n Þyk for all unit vectors x, y2c n. Thus, ðuau þ VBV Þx 6¼ 0 for all unit vector x2c n. So, rankðuau þ VBV Þ¼n for all unitary U, V2M n. Conversely, suppose rankðuau þ VBV Þ¼kfor all unitary U, V2M n. Assume that neither A nor B is scalar. If k < n then by Theorem 2.1, there is such that rankða I n ÞþrankðB þ I n Þ¼k. Since neither A nor B is a scalar, rankða I n Þ<k and rankðb þ I n Þ<k. By Proposition 2.5, there are unitary matrices U, V2M n such that rankðuau þ VBV Þ<k, which is a contradiction. Thus, n ¼ k. By Theorem 2.9, condition (b) holds. g

11 Ranks and determinants of the sum of matrices Determinants Let A, B2M n with eigenvalues a 1,..., a n, and b 1,..., b n, respectively. In this section we study the properties of ða, BÞ and P(A, B). For notational convenience and easy description of the results and proofs, we consider the sets DðA, BÞ ¼ðA, BÞ ¼fdetðX YÞ: X 2UðAÞ, Y 2UðBÞg and ( ) QðA, BÞ ¼PðA, BÞ ¼ Yn a j b ð j Þ : is a permutation of f1,..., ng : j¼1 It is easy to translate the results on D(A, B) and Q(A, B) to those on ða, BÞ and P(A, B), and vice versa. For any permutation ðð1þ,..., ðnþþ of ð1,..., nþ, there are unitary matrices U and V such that UAU* and VBV* are upper triangular matrices with diagonal entries a 1,..., a n and b ð1þ,..., b ðnþ, respectively. It follows that QðA, BÞ DðA, BÞ: The elements in Q(A, B) are called -points. Note also that if we replace (A, B) byðuau I n, VBV I n Þ for any 2C and unitary U, V2M n, the sets Q(A, B) and D(A, B) will be the same. Moreover, DðB, AÞ ¼ð 1Þ n DðA, BÞ and QðB, AÞ ¼ð 1Þ n QðA, BÞ. The following result can be found in [9]. THEOREM 3.1 Suppose A, B2M 2 have eigenvalues 1, 2 and 1, 2, respectively, and suppose A ðtr A=2ÞI 2 and B ðtr B=2ÞI 2 have singular values ab0 and cd0. Then DðA, BÞ is an elliptical disk with foci ð 1 1 Þð 2 2 Þ and ð 1 2 Þð 2 1 Þ with length of minor axis equal to 2ðac bd Þ. Consequently, D(A, B) is a singleton if and only if A or B is a scalar matrix; D(A, B) is a nondegenerate line segment if and only if A and B are nonscalar normal matrices. In the subsequent discussion, let WðAÞ ¼ x Ax: x 2 C n, x x ¼ 1 be the numerical range of A2M n Matrices whose determinantal ranges have empty interior THEOREM 3.2 Let A, B2M n with n3. Then DðA, BÞ ¼fg if and only if one of the following holds. (a) ¼ 0, and there is 2 C such that rank ða I n Þþrank ðb I n Þ < n. (b) 6¼ 0, one of the matrices A or B is a scalar matrix, and detða BÞ ¼.

12 116 C.-K. Li et al. Proof If (a) or (b) holds, then clearly D(A, B) is a singleton. If DðA, BÞ ¼f0g, then condition (a) holds by Corollary 2.3. Suppose DðA, BÞ ¼fgwith 6¼ 0. We claim that A or B is a scalar matrix. Suppose A and B have eigenvalues a 1,..., a n and b 1,..., b n, respectively. Assume that A has at least two distinct eigenvalues a 1, a 2 and B also has two distinct eigenvalues b 1, b 2. Then Q n j¼1 ða j b j Þ and ða 1 b 2 Þða 2 b 1 Þ Q n j¼3 ða j b j Þ will be two distinct -points, which is a contradiction because QðA, BÞDðA, BÞ is also a singleton. So, we have a 1 ¼¼a n or b 1 ¼¼b n. We may assume that the latter case holds; otherwise, interchange the roles of A and B. Suppose neither A nor B is a scalar matrix. Applying a suitable unitary similarity transform to B, we may assume that B is in upper triangular form with nonzero (1, 2) entry. Also, we may assume that A is in upper triangular form so that the leading two-by-two matrix is not a scalar matrix. Let A ¼ A 11 A 12 0 A 22 and B ¼ B 11 B 12 0 B 22 with A 11, B 11 2M 2, then DðA 11, B 11 Þ is a nondegenerate circular disk by Theorem 3.1. Since fdetða 22 B 22 Þ: 2 DðA 11, B 11 Þg DðA, BÞ, we see that D(A, B) cannot be a nonzero singleton. THEOREM 3.3 Suppose A, B2M n are such that D(A, B) is not a singleton. The following conditions are equivalent. (a) D(A, B) has empty interior. (b) D(A, B) is a nondegenerate line segment. (c) Q(A, B) is not a singleton, i.e., there are at least two distinct -points, and one of the following conditions holds. (c.1) A and B are normal matrices with eigenvalues lying on the same straight line or the same circle. (c.2) There is 2C such that one of the matrices A I n or B I n is rank 1 normal, and the other one is invertible normal so that the inverse matrix has collinear eigenvalues. (c.3) There is 2C such that A I n is unitarily similar to A ~ 0 n k and B I n is unitarily similar to 0 k ~B so that ~A2M k and ~B2M n k are invertible. In [3], the authors conjectured that for normal matrices A, B2M n,ifd(a, B) is contained in a line L, then L must pass through the origin. Using the above result, we see that the conjecture is not true. For example, if A ¼ diagð1, 1 þ i,1 iþ 1 and B ¼ diagð 1, 0, 0Þ, then D(A, B) is a straight line segment joining the points 1 i=2 and 1 þ i=2; see Corollary This shows that D(A, B) can be a subset of a straight line not passing through the origin. Of course, Theorem 3.3 covers more general situations. In (c.1), the line segment D(A, B) and the origin are collinear; in (c.3) the line segment D(A, B) has endpoints 0 and ð 1Þ n k detð ~AÞ detð ~BÞ; in (c.2) the line segment and the origin may or may not be collinear. g

13 Ranks and determinants of the sum of matrices 117 Since DðA, BÞ ¼ð 1Þ n DðB, AÞ, D(A, B) has empty interior (is a line segment) if and only if D(B, A) has empty interior (is a line segment). This symmetry will be used in the following discussion. We establish several lemmas to prove the theorem. Given a, b, c, d2c, with ad bc 6¼ 0, let fðzþ ¼ðaz þ bþ=ðcz þ dþ be the fractional linear transform on Cnf d=cg (C, ifc ¼ 0). If A2M n such that ca þ di n is invertible, one can define fðaþ ¼ðaA þ bi n ÞðcA þ di n Þ 1. The following is easy to verify. LEMMA 3.4 Suppose A, B2M n, and fðzþ ¼ðaz þ bþ=ðcz þ d Þ is a fractional linear transform such that f(a) and f(b) are well defined. Then Dð fðaþ, fðbþþ ¼ detððca þ di n ÞðcB þ di n ÞÞ 1 ðad bcþ n DðA, BÞ: LEMMA 3.5 Let A, B2M n with eigenvalues a 1,..., a n and b 1,..., b n, respectively. If D(A, B) has empty interior, then DðA, BÞ ¼DðA, diagðb 1,..., b n ÞÞ ¼ Dðdiagða 1,..., a n Þ, BÞ ¼ Dðdiagða 1,..., a n Þ, diagðb 1,..., b n ÞÞ: Proof Assume that D(A, B) has empty interior. Applying a unitary similarity transform we can assume that A ¼ða rs Þ and B ¼ðb rs Þ are upper triangular matrices. For any unitary matrix V2M n, let D ¼ðd rs Þ¼VBV. For any 2½0, 2Þ, let U ¼½e i ŠI n 1. Denote by X rs the ðn 1Þðn 1Þ matrices obtained from X2M n by deleting its r-th row and the s-th column. Expanding the determinant detðu AU DÞ along the first row yields detðu AU DÞ ¼ða 11 d 11 Þ detða 11 D 11 Þþ Xn ð 1Þ jþ1 ðe i a 1j d 1j Þ detða 1j D 1j Þ j¼2 " # ¼ ða 11 d 11 Þ detða 11 D 11 Þ Xn ð 1Þ jþ1 d 1j detða 1j D 1j Þ þ e i, j¼2 where ¼ P n j¼2 ð 1Þ jþ1 a 1j detða 1j D 1j Þ: Thus, CðA, VBV Þ¼fdetðU AU VBV Þ: 2½0, 2Þg is a circle with radius jj. Suppose jj 6¼ 0, i.e., CðA, VBV Þ is a nondegenerate circle. Repeating the construction in the previous paragraph on V ¼ I n, we get a degenerate circle CðA, BÞ ¼fdetðA BÞg: Since the unitary group is path connected, there is a continuous function t V t for t2½0, 1Š so that V 0 ¼ V and V 1 ¼ I n. Thus, we have a family of circles CðA, V t BV t Þ

14 118 C.-K. Li et al. in D(A, B) transforming CðA, VBV Þ to C(A, B). Hence, all the points inside CðA, VBV Þ belong to D(A, B). Thus, D(A, B) has nonempty interior. As a result, ¼ Xn j¼2 ð 1Þ jþ1 a 1j detða 1j D 1j Þ¼0, and detða VBV Þ¼detðA DÞ ¼ða 11 d 11 Þ detða 11 D 11 Þ Xn ð 1Þ jþ1 d 1j detða 1j D 1j Þ ¼ detða 1 DÞ¼detðA 1 VBV Þ, j¼2 where A 1 is the matrix obtained from A by changing all the nondiagonal entries in the first row to zero. It follows that DðA, BÞ ¼DðA 1, BÞ. Inductively, by expanding the determinant detðu j A j U j DÞ along the ( j þ 1)-th row with U j ¼ I j ½e i ŠI n j 1, we conclude that DðA j, BÞ ¼DðA jþ1, BÞ where A jþ1 is the matrix obtained from A j by changing all the nondiagonal entries in the ð j þ 1Þ-th row to zero. Therefore, DðA, BÞ ¼DðA 1, BÞ ¼DðA 2, BÞ ¼¼DðA n 1, BÞ ¼Dðdiagða 11,..., a nn Þ, BÞ: Note that a 11,..., a nn are the eigenvalues of A as A is in the upper triangular form. Similarly, we can argue that DðA, BÞ ¼DðA, diagðb 1,..., b n ÞÞ. Now, we can apply the argument to Dðdiagða 1,..., a n Þ, BÞ to get the last set equality. g LEMMA 3.6 Let A ¼ ^A 0 n k, where ^A2M k with k2f1,..., n 1g is upper triangular invertible. If B2M n has rank n k, then DðA, BÞ ¼ fð 1Þ n k detð ^AÞ detðx BXÞ: X is n ðn kþ, X X ¼ I n k g: If B ¼ 0 k ^B so that ^B2M n k is invertible, then D(A, B) is the line segment joining 0 and ð 1Þ n k detð ^AÞ detð ^BÞ. Proof Suppose A ¼ða rs Þ has columns A 1,..., A n, and U BU has columns B 1,..., B n. Let C be obtained from A U BU by removing the first column, and let B 22 be obtained from C by removing the first row. By linearity of the determinant function on the first column, detða U BUÞ ¼detð½A 1 jc ŠÞ detð½b 1 jc ŠÞ ¼ a 11 detðb 22 Þþ0, because ½B 1 jc Š has rank at most n 1. Inductively, we see that detða U BUÞ ¼ð 1Þ n k detð ^AÞ detðyþ where Y is obtained from U BU by removing its first k rows and first k columns.

15 Ranks and determinants of the sum of matrices 119 Now if B ¼ 0 k ^B so that ^B2M n k is invertible, then the set fdetðx BXÞ: X X ¼ I n k g is a line segment joining 0 and detð ^BÞ; e.g., see [6]. Thus, the last assertion follows. g LEMMA 3.7 Suppose A and B are not both normal such that A B has exactly n nonzero eigenvalues. If D(A, B) has no interior point, then there exist 2C and 0kn such that A I n and B I n are unitarily similar to matrices of the form ~A 0 n k and 0 k ~B for some invertible matrices ~A2M k and ~B2M n k. Proof Suppose A or B is a scalar matrix, say B ¼ I n. Under the given hypothesis, A I n is invertible and B I n ¼ 0. Thus, the result holds for k ¼ n. In the rest of the proof, assume that neither A nor B is a scalar matrix. We may assume that A ¼ða rs Þ¼ A 11 A 12 0 A 22 and B ¼ðb rs Þ¼ B 11 B 12 0 B 22 ð3:1þ such that A 11, B 11 2M m and A 22, B 22 2M n m are upper triangular matrices so that A 11, B 22 are invertible, and A 22, B 11 are nilpotent. If m ¼ 0, then A ¼ A 11 is nilpotent and B ¼ B 22 is invertible. We are going to show that A ¼ 0. Hence, the lemma is satisfied with k ¼ 0. Suppose A 6¼ 0. We may assume that a 12 6¼ 0. Let X ¼ 0 a and Y ¼ b 11 b 12 : 0 b 22 Since B is not a scalar matrix, we may assume that Y is not a scalar matrix either. Then D(X, Y) is a nondegenerate elliptical disk and ð 1Þ n detðbþ b 11 b 22 : 2 DðX, Y Þ DðA, BÞ: Therefore, D(A, B) has nonempty interior, a contradiction. Similarly, if m ¼ n, then B ¼ 0. Hence, we may assume that 1m<n in the following. We are going to show that A 12 ¼ 0 ¼ B 12 in (3.1). To this end, let X, Y2M 2 be the principal submatrices of A and B lying in rows and columns m and m þ 1. If a m, mþ1 6¼ 0orb m, mþ1 6¼ 0, then ða m,m b mþ1, mþ1 Þ 1 detða BÞDðX, Y Þ is an elliptical disk in D(A, B), which is impossible. Next, we show that a m 1, mþ1 ¼ 0 ¼ b m 1, mþ1. If it is not true, let X, Y2M 2 be the principal submatrices of A and B lying in rows and columns m 1 and m þ 1. For any unitary U, V2M 2, let ¼ detðuxu VYV Þ. Construct ^U (respectively, ^V ) from I n by changing the principal submatrix at rows and columns m 1 and m þ 1byU (respectively, V ). Then ^UA ^U is still in upper triangular block form so that its leading ðm 2Þðm 2Þ principal submatrix and its trailing ðn m 1Þðn m 1Þ principal

16 120 C.-K. Li et al. submatrix are the same as A. Moreover, since we have shown that a m, mþ1 ¼ 0 ¼ b m, mþ1, the principal submatrix of ^UA ^U lying in rows m 1, m, m þ 1 has the form 0 1 B 0 a mm 0 A: A similar result is true for ^VB ^V. Hence, detð ^UA ^U ^VB ^V Þ¼ detða BÞ=ða m 1, m 1 b mþ1, mþ1 Þ: As a result, D(A, B) contains the set ða m 1, m 1 b mþ1, mþ1 Þ 1 detða BÞDðX, YÞ, which is an elliptical disk. This is a contradiction. Next, we can show that a m 2, mþ1 ¼ 0 ¼ b m 2, mþ1 and so forth, until we show that a 1, mþ1 ¼ 0 ¼ b 1, mþ1. Note that it is important to show that a j, mþ1 ¼ 0 ¼ b j, mþ1 in the order of j ¼ m, m 1,..., 1. Remove the ðm þ 1Þ-th row and column from B and A to get ^B and ^A. Then a mþ1, mþ1 Dð ^A, ^BÞ is a subset of D(A, B) and has no interior point. An inductive argument shows that the (1, 2) blocks of A and B are zero. Thus, A 12 ¼ 0 ¼ B 12. If B 11 and A 22 are both equal to zero, then the desired conclusion holds. Suppose B 11 or A 22 is nonzero. By symmetry, we may assume that B 11 6¼ 0. CLAIM (1) A 11 ¼ I m and (2) B 22 ¼ I n m for some 6¼ 0. If this claim is proved, then A I n ¼ 0 m ða 22 I n m Þ and B I n ¼ ðb 11 I m Þ0 n m. Thus, the result holds with k ¼ n m. To prove our claim, suppose B 11 6¼ 0. Then m2 and we may assume that its leading 22 submatrix B 0 is a nonzero strictly upper triangular matrix. If A 11 is non-scalar, then we may assume that its leading 22 submatrix A 0 is non-scalar. But then DðA 0, B 0 Þ will generate an elliptical disk in D(A, B), which is impossible. So, A 11 ¼ I m for some 6¼ 0. This proves (1). Now we prove (2). Suppose B 22 6¼ I m n. Thus, n m2 and we may assume that 44 submatrices of A and B lying at rows and columns labeled by m 1, m, m þ 1, m þ 2 have the forms A 0 ¼ I 2 0 and B 0 ¼ B B0 2 with 2C, a nonzero 22 nilpotent matrix B 0 1 and a 22 matrix B0 2 such that B0 2 6¼ I 2. Let P ¼½1Š 0 1 ½1Š, V ¼ V 1 V with unitary V 1, V 2 2M 2. Then detðpa 0 P t VB 0 V Þ¼ 1 2 with 1 ¼ detðdiagð,0þ V 1 B 0 1 V 1 Þ and 2 ¼ detðdiagð,0þ V 2 B 0 2 V 2 Þ:

17 Ranks and determinants of the sum of matrices 121 Since B 0 2 6¼ I 2, by Theorem 3.1, we can choose some unitary V 2 such that 2 6¼ 0. Also as B 0 1 is nonzero nilpotent, by Theorem 3.1, one can vary the unitary matrices V 1 to get all values 1 in the nondegenerate circular disks Dðdiagð,0Þ, B 0 1Þ. Hence, ð 2 b mþ1, mþ1 b mþ2, mþ2 Þ 1 2 detða BÞ Dðdiagð,0Þ, B 0 1ÞDðA, BÞ so that D(A, B) also has nonempty interior, which is the desired contradiction. g LEMMA 3.8 Let A; B 2 M n be normal matrices. Then QðA; BÞ ¼fg if and only if one of the following holds. (a) ¼ 0, and A B has an eigenvalue with multiplicity at least n þ 1. (b) 6¼ 0, and one of the matrix A or B is a scalar matrix, and detða BÞ ¼. Proof Clearly if (a) or (b) holds, then QðA; BÞ is a singleton. Let A and B have eigenvalues a 1,...,a n and b 1,...,b n, respectively. Suppose QðA; BÞ ¼fg. If both A and B are not scalar matrices, then A has at least two distinct eigenvalues, say a 1, a 2 and B also has two distinct eigenvalues, say b 1 ; b 2. Then ¼ Q n i¼1 ða i b i Þ¼ða 1 b 2 Þða 2 b 1 Þ Q n i¼3 ða i b i Þ implies that ¼ 0. Now we claim that condition (a) holds. Suppose not, then every eigenvalue of A B has multiplicity at most n. For k ¼ 1; 2;...; n, let S k ¼fi : b i 6¼ a k g. Suppose 1 k 1 < k 2 < < k m n. Then i 62 [ m j¼1 S k j if and only if b i ¼ a k1 ¼ a k2 ¼¼a km. Therefore, there are at most n minot in [ m j¼1 S k j. Hence, [ m j¼1 S k j contains at least m elements. By the theorem of P. Hall [19], there exist i k 2 S k, k ¼ 1;...; n, with i k 6¼ i k 0 for k 6¼ k 0. Thus, Q n k¼1 ða k b ik Þ 6¼ 0, which contradicts the fact that ¼ 0. g LEMMA 3.9 Suppose A, B2M n are normal matrices such that A has at least three distinct eigenvalues, each eigenvalue of B has multiplicity at most n 2, and each eigenvalue of A B has multiplicity at most n 1. Then there are three distinct eigenvalues a 1, a 2, a 3 of A satisfying the following condition. For any eigenvalue b of B with b =2fa 1, a 2, a 3 g, there exist eigenvalues b 1, b 2, b 3 of B with b 1 =2fb 2, b 3 g, b 3 ¼ b, and the remaining eigenvalues can be labeled so that Q n j¼4 ða j b j Þ 6¼ 0. Moreover, if A has more than three distinct eigenvalues, and B has exactly two distinct eigenvalues, then we can replace a 3 by any eigenvalue of A different from a 1, a 2, a 3, and get the same conclusion. Proof Let A and B have k distinct common eigenvalues 1, 2,..., k so that j has multiplicity m j in the matrix A B for j ¼ 1,..., k, with m 1 m k. By our assumption, n 1m 1. The choices for a i and b i depend on k. We illustrate the different cases in the following table. k ¼ 0 k ¼ 1 k ¼ 2 k 3 a 1 * a 2 * * 2 2 a 3 * * * 3 b 1 6¼b b 2 6¼b 1 6¼b b 3 b b b b

18 122 C.-K. Li et al. where denotes any choice subject to the condition that a 1, a 2, a 3 are distinct eigenvalues of A. For any eigenvalue b of B with b =2fa 1, a 2, a 3 g, set b 3 ¼ b and choose b 1 ¼ 1 if k1 and b 1 to be any eigenvalue of B not equal to b. Since the multiplicity of b 1 is n 2, there is always a third eigenvalue b 2 of B, with b 2 6¼ b 1. Furthermore, we can choose b 2 ¼ 2 if k2. Use the remaining eigenvalues of A and B to construct the matrices ^A ¼ diagða 4,..., a n Þ and ^B ¼ diagðb 4,..., b n Þ. By Lemma 3.8, the proof will be completed if we can prove the following: CLAIM If is a common eigenvalue of ^A and ^B then the multiplicity of in the matrix ^A ^B is at most n 3. To verify our claim, let be a common eigenvalue of ^A and ^B. Then ¼ r with r2f1,..., kg. If r2f1, 2g, then two of the entries ða 1, a 2, a 3, b 1, b 2, b 3 Þ equals r by our construction. Since n 1m r, the multiplicity of r in ^A ^B equals m r 2n 3. If r ¼ 3, then b 3 6¼ i for i ¼ 1, 2, 3. Thus, m 3 ðm 1 þ m 2 þ m 3 Þ=3½ð2n 1Þ=3Šn 2, where ½tŠ is the integral part of the real number t. Since one of the entries in ða 1, a 2, a 3, b 1, b 2, b 3 Þ equals 3, we see that the multiplicity of 3 in ^A ^B equals m 3 1n 3. Suppose r ¼ 4. If n ¼ 4 then ða 1, a 2 Þ¼ðb 1, b 2 Þ¼ð 1, 2 Þ, a 3 ¼ 3 and b 3 ¼ 4 by our construction. Thus, a 4 b 4 ¼ 4 3 6¼ 0. If n5, then the multiplicity of r in ^A ^B is at most m 4 ½2n=4Šn 3. If r5, then nr5 and the multiplicity of r in ^A ^B is at most m r ð2n=5þn 3. By the above arguments, the claim holds. Note that if B has exactly two distinct eigenvalues then k2 and a 3 can be chosen to be any eigenvalue different from a 1, a 2 in our construction. Thus, the last assertion of the lemma follows. g LEMMA 3.9 Let A ¼ diagða 1, a 2, a 3 Þ and B ¼ diagðb 1, b 2, b 3 Þ with a j 6¼ a k for 1 j<k3 and b 1 6¼ b 2. Suppose D(A, B) has empty interior. Then a 1, a 2, a 3, b 3 are either concyclic or collinear. Proof By Lemma 3.4, we may apply a suitable fractional linear transform and assume that ða 1, a 2, a 3 Þ¼ða,1,0Þ with a2rnf0, 1g. By the result in [7], if U ¼ðu rs Þ2M 3 is unitary and S U ¼ðju rs j 2 Þ, then detðuau BÞ ¼detðAÞþð 1Þ 3 detðbþ ðb 1, b 2, b 3 ÞS U ð0, 0, aþ t þðb 2 b 3, b 1 b 3, b 1 b 2 ÞS U ða,1,0þ t : Let C ¼ða,1,0Þ t ðb 2 b 3, b 1 b 3, b 1 b 2 Þ ð0, 0, aþ t ðb 1, b 2, b 3 Þ: Then detðuau BÞ ¼detðAÞþð 1Þ 3 detðbþþtr ðcs U Þ:

19 Ranks and determinants of the sum of matrices 123 It follows that the set R¼ tr C ju rs j 2 : ðurs Þ is unitary has empty interior. Let S 0 be the 33 matrix with all entries equal to 1=3. For, 2½0, 1=5Š, let þ 1 1 þð Þ S ¼ Sð, Þ ¼ 3 þ ð Þ B C B 3 ¼ S 0 1 Að Þ: A Since 4 15 rffiffiffiffiffiffiffiffiffiffiffiffiffi 1 2, 9 rffiffiffiffiffiffiffiffiffiffiffiffiffi 1 2, 9 rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 ð Þ , by the result in [1], there is a unitary ðu rs Þ such that ðju rs j 2 Þ¼S. Direct calculation shows that trðcsþ ¼trðCS 0 Þþðb 1 b 2 Þ½ðab 3 þ aþ ðb 3 þ aþš: The set R having empty interior implies that ðab 3 þ aþ and ðb 3 þ aþ are linearly independent over reals, which is possible only when b 3 is real. Thus, fa 1, a 2, a 3, b 3 gr and the result follows. g Proof of Theorem 3.3 The implication (b) ) (a) is clear. Suppose (c) holds. If (c.1) holds, then D(A, B) is a line segment on a line passing through origin as shown in [3]. If (c.2) holds, then we can assume that A I n ¼ diagða,0,...,0þ, and B I n has full rank and the eigenvalues of ðb I n Þ 1 are collinear. We may replace (A, B) by ða I n, B I n Þ and assume that ¼ 0. Since B 1 is normal with collinear eigenvalues, the numerical range WðB 1 Þ of B 1 is a line segment. Let U2M n be unitary, and U BU ¼ b 11 B 22 with B 22 2M n 1. Then detðb 22 Þ is the (1, 1) entry of detðbþu B 1 U. Hence, detðb 22 Þ=detðBÞ2WðB 1 Þ. Thus, detðuau BÞ ¼detðA U BUÞ ¼að 1Þ n 1 detðb 22 Þþð 1Þ n detðu BUÞ 2 að 1Þ n 1 detðbþwðb 1 Þþð 1Þ n detðbþ:

20 124 C.-K. Li et al. If (c.3) holds, then D(A, B) is the line segment joining 0 and ð 1Þ n k detð AÞ ~ detð ~BÞ by Lemma 3.6. Thus, we have (c) ) (b). Finally, suppose (a) holds, i.e., D(A, B) has empty interior. Since D(A, B) is not a singleton, neither A nor B is a scalar matrix. Suppose A and B have eigenvalues a 1,..., a n and b 1,..., b n. Let A 0 ¼ diagða 1,..., a n Þ and B 0 ¼ diagðb 1,..., b n Þ. By Lemma 3.5, DðA 0, B 0 Þ¼DðA, BÞ and hence DðA 0, B 0 Þ is not singleton. It then follows from Corollary 2.2 and Lemma 3.8 that QðA 0, B 0 Þ is not a singleton, so as QðA, BÞ. Now we show that one of (c.1) (c.3) holds. Suppose A and B have k distinct common eigenvalues 1, 2,..., k such that j has multiplicity m j in the matrix A B for j ¼ 1,..., k, with m 1 m k. Since QðA, BÞ ¼QðA 0, B 0 Þ 6¼ f0g, we have m 1 n. If m 1 ¼ n, then ða 1 IÞðB 1 IÞ has exactly n nonzero eigenvalues, and hence (c.3) follows from Lemma 3.7. Suppose k ¼ 0orm j n 1 for all 1 jk. We claim that both A and B are normal. If it is not true, we may assume that A is not normal. Otherwise, interchange the roles of A and B. Then we may assume that A is in upper triangular form with nonzero (1, 2) entry by the result in [27]. Suppose A has diagonal entries a 1,..., a n. Let A 1 be the leading 22 principal submatrix of A. We can also assume that B is upper triangular with diagonal b 1,..., b n, where b 1 6¼ b 2 satisfies the following additional assumptions: (1) If fa 1, a 2 g\f 1,..., k g¼60, then b 1 and b 2 are chosen so that j 2fb 1, b 2 g for 1 jminfk,2g. (2) If fa 1, a 2 g\f 1,..., k g 6¼ 6 0, then b 1 and b 2 are chosen so that j 2fa 1, a 2, b 1, b 2 g for 1 jminfk,3g. Then b 3,..., b n can be arranged so that p ¼ Q n j¼3 a j b j 6¼ 0. It follows from Theorem 3.1 that fp: 2DðA 1, diagðb 1, b 2 ÞÞg is a nondegenerate elliptical disk in D(A, B), which is a contradiction. Now, suppose both A and B are normal, and assume that k ¼ 0orm j n 1 for all 1 jk. Case 1 Suppose A or B has an eigenvalue with multiplicity n 1. Interchanging the role of A and B, if necessary, we may assume that A ¼ diagða,0,...,0þþa 2 I n. We can further set a 2 ¼ 0; otherwise, replace (A, B) by ða a 2 I n, B a 2 I n Þ. Since m j n 1, we see that B is invertible. Moreover, for any unitary matrix U, let u be the first column of U, and let ~U be obtained from U by removing u. Then detða U BU Þ¼ð 1Þ n detðbþ adetð ~U B ~U Þ : Note that detð ~U B ~U Þ= detðbþ is the (1, 1) entry of ðu BUÞ 1, and equals u B 1 u. So, DðA, BÞ ¼ ð 1Þ n detðbþ adetðbþu B 1 u : u 2 C n, u u ¼ 1 : Since D(A, B) is a set with empty interior and so is the numerical range WðB 1 Þ of B 1. Thus, B 1 has collinear eigenvalues; see [21]. Hence condition (c.2) holds.

21 Ranks and determinants of the sum of matrices 125 Case 2 Suppose both A and B have two distinct eigenvalues and each eigenvalue of A and B has multiplicity at most n 2. Let A and B have two distinct eigenvalues, say, a 1, a 2 and b 1, b 2, respectively. We claim that a 1, a 2, b 1, b 2 are on the same straight line or circle, i.e., condition (c.1) holds. Suppose it is not true. Assume that a 1, a 2 and b 1 are not collinear and b 2 is not on the circle passing through a 1, a 2 and b 1. Then there is a factional linear transform f(z) such that f(a) and f(b) has eigenvalues 1, 0 and a, b, respectively, where a2r nf1, 0g and b =2 R. By Lemma 3.4, D(A, B) has empty interior if and only if Dð fðaþ, fðbþþ has empty interior. We may replace (A, B) byð fðaþ, fðbþþ and assume that A ¼ diagð1, 0, 1, 0ÞA 2, B ¼ diagða, b, a, bþb 2 with detða 2 B 2 Þ 6¼ 0. By Theorem 3.1, Dðdiagð1, 0Þ, diagða, bþþ ¼ ð1 sþaðb 1Þþsbða 1Þ: s 2½0, 1Š ¼ aðb 1Þþsða bþ: s 2½0, 1Š : Hence, D(A, B) contains the set R¼ detða 2 B 2 Þðaðb 1Þþsða bþþðaðb 1Þþtða bþþ: s, t 2½0, 1Š (! ) ¼ detða 2 þ B 2 Þðaðb 1ÞÞ 2 a b 1 þðsþtþ aðb 1Þ þ st a b 2 : s, t 2½0, 1Š : aðb 1Þ Note that fðst, s þ tþ: s, t2½0, 1Šg has nonempty interior. Let r ¼ða bþ=aðb 1Þ. Then ðar þ 1Þb ¼ að1 þ rþ and so r cannot be real. Therefore, the complex numbers r and r 2 are linearly independent over reals. Hence the mapping ðu, vþ 1 þ ur þ vr 2 is an invertible map from R 2 to C. Thus, the set RDðA, BÞ has nonempty interior, which is a contradiction. Case 3 Suppose each eigenvalue of A and B has multiplicity at most n 2 and one of the matrices has at least three distinct eigenvalues. Assume that A has at least three distinct eigenvalues. Otherwise, interchange the roles of A and B. By Lemma 3.8, there are three distinct eigenvalues of A, say, a 1, a 2, a 3, such that the conclusion of the lemma holds. Applying a fractional linear transformation, if necessary, we may assume that a 1, a 2, a 3 are collinear. For any eigenvalue b of B with b =2fa 1, a 2, a 3 g we can get b 1 ; b 2 and b 3 ¼ b satisfying the conclusion of Lemma 3.8. Therefore, D(A, B) contains the set f Q n j¼4 ða j b j Þ: 2Dðdiagða 1, a 2, a 3 Þ, diagðb 1, b 2, b 3 ÞÞg. Since D(A, B) has empty interior, Lemma 3.9 ensures that a 1, a 2, a 3 and b are collinear. Therefore, all eigenvalues of B lie on the line L passing through a 1, a 2, a 3. Suppose B has three distinct eigenvalues. We can interchange the roles of A and B and conclude that the eigenvalues of A lie on the same straight line L. Suppose B has exactly two eigenvalues, and a is an eigenvalue of A with a =2fa 1, a 2, a 3 g such that a is not an eigenvalue of B. By Lemma 3.8, we may replace a 3 by a and show that a 1, a 2, a and the two eigenvalues of B belong to the same straight line. Hence, all eigenvalues of A and B are collinear and (c.1) holds in this case. g

22 126 C.-K. Li et al Sharp points A boundary point of a compact set S in C is a sharp point if there exists d>0 and 0t 1 <t 2 < t 1 þ such that S\fz 2 C: jz j dg f þ e i : 2½0, dš, 2½t 1, t 2 Šg: It was shown [3, Theorem 2] that for two normal matrices A, B2M n such that the union of the spectra of A and B has 2n distinct elements, a nonzero sharp point of D(A, B) is a -point, that is, an element in Q(A, B). More generally, we have the following. THEOREM 3.10 Let A, B2M n. Every sharp point of D(A, B) is a -point. Proof Using the idea in [2], we can show that a nonzero sharp point detðuau BÞ is a -point as follows. For simplicity, assume U ¼ I n so that detða BÞ is a sharp point of D(A, B). For each Hermitian H2M n, consider the following one parameter curve in D(A, B): det A e ih Be ih ¼ detða BÞ 1 þ itr ðða BÞ 1 ½H, BŠÞ þ O 2, where ½X, YŠ ¼XY YX. Since detða BÞ is a sharp point, and hence 0 ¼ trða BÞ 1 ½H, BŠ ¼tr H½B, ða BÞ 1 Š for all Hermitian H, 0 ¼½B, ða BÞ 1 Š¼BðA BÞ 1 ða BÞ 1 B: Consequently, 0 ¼ðA BÞB BðA BÞ, equivalently, AB ¼ BA. Thus, there exists a unitary V such that both VAV* and VBV* are in triangular form. As a result, detða BÞ ¼detðVðA BÞV Þ is a -point. Next, we refine the previous argument to treat the case when detða BÞ ¼0is a sharp point. If the spectra of A and B overlap, then 0 is a -point. So, we assume that A and B has disjoint spectra. By Theorem 2.7, there is unitary U such that UAU B has rank n 1. Assume U ¼ I n so that A B has rank n 1 and detða BÞ ¼0 is a sharp point. Then for any Hermitian H and 1kn, det A e ih Be ih ¼ det A B þ iðhb BHÞþ 2 M! ¼ detða BÞþi Xn r kj s jk þ O 2, where adjða BÞ ¼R ¼ðr pq Þ and HB BH ¼ S ¼ðs pq Þ. Thus, for k ¼ 1,..., n we have P n j¼1 r kjs jk ¼ 0, and hence j¼1 0 ¼ tr RS ¼ trðadjða BÞðHB BHÞÞ ¼ tr H ½BadjðA BÞ adjða BÞBŠ for every Hermitian H. So, B and adjða BÞ commute. Since A B has rank n 1, the matrix adjða BÞ has rank 1, and equals uv t for some nonzero vectors u, v. Comparing the columns of the matrices on left and right sides of the equality Buv t ¼ uv t B, we see that Bu ¼ bu for some b2c. Similarly, we have Auv t ¼ uv t A and hence Au ¼ au for

23 some a2c. Consequently, 0 ¼ðA BÞadjðA BÞ ¼ðA BÞuv t ¼ða bþuv t. Thus, a b ¼ 0, i.e., the spectra of A and B overlap, which is a contradiction. g Clearly, if D(A, B) is a line segment, then the end points are sharp points. By Theorem 3.3 and the above theorem, we have the following corollary showing that Marcus Oliveira conjecture holds if D(A, B) has empty interior. COROLLARY 3.11 Let A, B2M n.ifd(a, B) has empty interior, then D(A, B) equals the convex hull of Q(A, B). By Theorem 2.9, 02DðA, BÞ if for every 2C, the singular values of A I n and B I n do not lie in two disjoint closed intervals in ½0, 1Þ. Following is a sufficient condition for A, B2M n to have 0 as a sharp point of D(A, B) in terms of W(A) and W(B). PROPOSITION 3.12 Ranks and determinants of the sum of matrices 127 Let A, B2M n be such that 02DðA, BÞ and Then Proof WðAÞ[Wð BÞ re i : r 0, 2ð =ð2nþ, =ð2nþþ : DðA, BÞ re i : r 0, 2ð =2, =2Þ : Note that for any unitary U and V, there is a unitary matrix R such that RðUAU VBV ÞR ¼ a pq bpq is in upper triangular form. Hence, a pp b qq ¼ r p e i p with r p 0 and p 2ð =ð2nþ, =ð2nþþ for p ¼ 1,..., n. So, detðuau VBV Þ¼ Q n p¼1 ða pp b pp Þ¼re i with r0 and 2ð =2, =2Þ. g 4. Further extensions There are many related topics and problems which deserve further investigation. One may ask whether the results can be extended to the sum of k matrices from k different unitary orbits for k>2. For the norm problem, in an unpublished manuscript Li and Choi have extended the norm bound result to k matrices A 1,..., A k 2M n for k2 to max kx 1 þþx k k: X j 2UðA j Þ, j ¼ 1,..., k ( ) ¼ min Xk ka j j I n k: j 2 C, j ¼ 1,..., k, Xk j ¼ 0 : j¼1 However, we are not able to extend the maximum rank result in section 2 to k matrices with k>2 at this point. In any event, it is easy to show that for any 1,..., k 2C satisfying P k j¼1 j ¼ 0, min rankðx 1 þþx k Þ: X j 2UðA j Þ, j ¼ 1,..., k max rankðaj j I n Þ : j ¼ 1,..., k : j¼1

24 128 C.-K. Li et al. It is challenging to determine all the possible rank values of matrices in UðA 1 ÞþþUðA k Þ. For Hermitian matrices A 1,..., A k, there is a complete description of the eigenvalues of the matrices in UðA 1 ÞþþUðA k Þ; see [16]. Evidently, the set ða 1,..., A k Þ¼ ( ) det X k j¼1 X j!: X j 2UðA j Þ, j ¼ 1,..., k is a real line segment. When k ¼ 2, the end points of the line segment have the form detðx 1 þ X 2 Þ for some diagonal matrices X 1 2UðA 1 Þ and X 2 2UðA 2 Þ; see [15]. However, this is not true if k>2 as shown in the following example. Example 4.1 Let A ¼ 3 0, B ¼ , C ¼ Then for any unitary U, V, W2M 2, the matrix UAU þ VBV þ WCW is positive definite with eigenvalues 7 þ d and 7 d with d2½0, 7Þ. Hence detðuau þ VBV þ WCW Þ7 2 ¼ detða þ B þ CÞ: Thus, the right end point of the set ða, B, CÞ is not of the form ða 1 þ b ð1þ þ c ð1þ Þða 2 þ b ð2þ þ c ð2þ Þ for permutations and of (1, 2). It is interesting to determine the set ða 1,..., A k Þ for Hermitian, normal, or general matrices A 1,..., A k 2M n. Inspired by the Example 4.1, we have the following observations. (1) Suppose A 1,..., A k are positive semi-definite matrices. If there are unitary U 1,..., U k such that X k j¼1 U j AU j ¼ I n : ð4:1þ for some scalar, then max ða 1,..., A k Þ¼ n. The necessary and sufficient conditions for (4.1) to hold can be found in [15]. (2) Let A 1, A 2, A 3 be Hermitian matrices such that detða 1 þ A 2 þ A 3 Þ¼ max ða 1, A 2, A 3 Þ. Then there exist unitary U and V such that UA 1 U and VðA 2 þ A 3 ÞV are diagonal and detðua 1 U þ VðA 2 þ A 3 ÞV Þ¼detðA 1 þ A 2 þ A 3 Þ: Proof Let U be unitary matrix such that UA 1 U ¼ D 1 is diagonal. Suppose A 2 þ A 3 has eigenvalues 1,..., n. By the result of [15], there exists a permutation matrix P such that detðd 1 þ P diagð 1,..., n ÞP ÞdetðA 1 þða 2 þ A 3 ÞÞ: