On spectra and Brown s spectral measures of elements in free products of matrix algebras

Transcription

1 arxiv:math/06117v4 [mathoa] 8 Jul 007 On spectra and Brown s spectral measures of elements in free products of matrix algebras Junsheng Fang Don Hadwin Xiujuan Ma Abstract We compute spectra and Brown measures of some non self-adjoint operators in M C, 1 Tr M C, 1 Tr, the reduced free product von Neumann algebra of M C with M C Examples include AB and A + B, where A and B are matrices in M C, 1 Tr 1 and 1 M C, 1 Tr, respectively We prove that AB is an R-diagonal operator in the sense of Nica and Speicher [1] if and only if TrA TrB 0 We show that if X AB or X A + B and A,B are not scalar matrices, then the Brown measure of X is not concentrated on a single point By a theorem of Haagerup and Schultz [9], we obtain that if X AB or X A + B and X λ1, then X has a nontrivial hyperinvariant subspace affiliated with M C, 1 Tr M C, 1 Tr Keywords: free products, spectrum, Brown measure, R-diagonal operators, hyperinvariant subspaces 1 Introduction In 1983, LG Brown [] introduced a spectral distribution measure for non-normal elements in a finite von Neumann algebra with respect to a fixed normal faithful tracial state, which is called the Brown measure of the operator Recently, U Haagerup and H Schultz [9] proved a remarkable result which states that if the support of Brown measure of an operator in a type II 1 factor contains more than two points, then the operator has a non-trivial hyperinvariant subspace affiliated with the type II 1 factor In general cases, the computation of Brown measures of non-normal operators are nontrivial The first essential result was given by Haagerup and F Larsen In [8], Haagerup and Larsen computed the spectrum and Brown measure of R-diagonal operators in a finite von Neumann algebra, in terms of the distribution of its radial part Brown measures of some non-normal and non-r-diagonal operators, examples include u n + u, where u n and u are the generators of Z n and Z respectively, in the free product Z n Z, and elements of the form S α + is β, where S α and 1

2 S β are free semi-circular elements of variance α and β, are computed by P Biane and F Lehner in [1] The purpose of this paper is to compute the spectra and Brown measures of some non hermitian operators in M C, 1Tr M C, 1 Tr, the reduced free product von Neumann algebra of M C with M C cf [Ch] Examples include AB and A + B, where A and B are matrices in M C, 1Tr 1 and 1 M C, 1 Tr, respectively This paper is organized as follows In section we recall preliminary facts about Brown measures, R-diagonal operators, Haagerup and Larsen s result on Brown measures of R-diagonal operators and some notation used in this paper In section 3, we provide some results on the spectra and spectral radius of operators in M C M C, the universal free product C*-algebra of M C with M C Firstly we compute the spectral radius of AB for two normal matrices A M C 1 and B 1 M C relative to M C M C As a corollary, we also get the spectrum radius of AB for normal matrices A M C, 1 Tr 1 and B 1 M C, 1 Tr, relative to the reduced free product von Neumann algebra of M C with M C Then we obtain the following result: Let A,B be matrices in M C 1 and 1 M C, respectively, such that TrA TrB 0 Then σab, the spectrum of AB, relative to M C M C, is the closure of the annulus centered at 0 with inner radius A 1 1 B 1 1 and outer radius A B, where we use the convention 1 0 and if A is not invertible then A 1 : In section 4 we prove that AB is an R-diagonal operator if and only if TrA TrB 0, where A M C, 1Tr 1 and B 1 M C, 1 Tr As a corollary, we explicitly compute the spectrum and Brown measure of AB TrA TrB 0 in terms of transform of A A and B B In section 5, we develop algebraic techniques used in [4] Let X 1 M C, 1Tr With respect to the matrix units of M C, 1Tr 1, X x1 x By [4], M x 3 x C, 1Tr 4 M C, 1Tr LF 3 M C So x 1, x, x 3, x 4 LF 3 In section 5, we find -free generators h, u, v of LF 3 different from the free generators given in [4] so that we may explicitly write out x 1, x, x 3, x 4 in terms of h, u, v In section 6, we compute miscellaneous examples of Brown measures of operators A+ B and AB, where A M C, 1Tr 1 and B 1 M C, 1 Tr As a corollary, we show that A + B is an R-diagonal operator if and only if A + B 0 In section 7, we prove the following result: Let A M C, 1 Tr 1 and B 1 M C, 1 Tr if X A + B or X AB and A, B are not scalar matrices, then the Brown measure of X is not concentrated on a single point As a corollary of Theorem

3 3 71 of [H-S1], we prove that if X A+B or X AB and X λ1, then X has a nontrivial hyperinvariant subspace affiliated with M C, 1 Tr M C, 1 Tr Many concrete examples of spectra and Brown measures are given in this paper For some interesting applications, we refer to [5] Preliminaries 1 Fuglede-Kadison determinant and Brown s spectral measure Let M be a finite von Neumann algebra with a faithful tracial state τ The Fuglede-Kadison determinant [6], : M [0, [, is given by T exp{τlog T }, T M, with exp{ } : 0 For an arbitrary element T in M the function λ log a λ1 is subharmonic on C, and it s Laplacian dµ T λ : 1 π log T λ1, in the distribution sense, defines a probability measure µ T on C, called the Brown s measure [] of T From the definition, Brown measure µ T only depends on the joint distribution of T and T If T is normal, µ T is the trace τ composed with the spectral projections of T If M M n C and τ 1 n Tr is the normalized trace on M nc, then µ T is the normalized counting measure 1 n δ λ 1 + δ λ + δ λn, where λ 1, λ, λ n are the eigenvalues of T repeated according to root multiplicity The Brown measure has the following properties see [, 10]: µ T is the unique compactly supported measure on C such that log T λ1 log z λ dµ C Tz for all λ C The support of µ T is contained in σt, the spectrum of T µ ST µ TS for arbitrary S, T in M, and if fz is analytic in a neighborhood of σa, µ ft µ T f, the push-forward measure of µ T under the map λ fλ If E M is a projection such that E LatT, then with respect to E, I E we can write T A B 0 C,

4 4 where A ETE and C I ETI E are elements of M 1 EME and M I EMI E, respectively Let µ A and µ C be the Brown measures of A and C computed relative to M 1 and M, respectively Then µ T αµ A +1 αµ C, where α τe For a generalization of Brown measures of sets of commuting operators in a type II 1 factor, we refer to [15] R-diagonal operators In 1995, A Nica and S Speicher [1] introduced the class of R-diagonal operators in noncommutative probability spaces Recall that an operator T in a non-commutative probability space is an R diagonal operator if the R transform R µt,t of the joint distribution µt, T of T, T is of the form R µt,t z 1, z α n z 1 z n + n1 α n z z 1 n Nica and Speicher [1] proved that T is an R diagonal operator if and only if T has same -distribution as product UH, where U and H are -free random variables in some tracial non commutative probability space, U is a Haar unitary operator and H is positive If T is an R-diagonal operator, then the -distribution of T is uniquely determined by the distribution of T T T If T is an R-diagonal operator and S is -free with T, then both ST and T S are R-diagonal operators see [1] If T is an R-diagonal operator and n N, then T n is also an R-diagonal operator see [8, 11] For other important properties of R-diagonal operators, we refer to [8, 11, 1, 13] 3 Brown measures of R-diagonal operators In [8], Haagerup and Larson explicitly computed the Brown measures of R-diagonal operators in a finite von Neumann algebra Theorem 1 Theorem 44 of [8] Let U, H be -free random variables in a noncommutative probability space M, τ, with U a Haar unitary operator and H a positive operator such that the distribution µ H of H is not a Dirac measure Then the Brown measure µ UH of UH can be computed as the following 1 µ UH is rotation invariant and its support is the annulus with inner radius H 1 1 and outer radius H µ UH {0} µ H {0} and for t ]µ H {0}, 1], µ UH B 0, t 1 1/ µh t, n1

5 5 where µh is the transform of H and B0, r is the open disc with center 0 and radius r; 3 µ UH is the only rotation invariant symmetric probability measure satisfying Furthermore, if H is invertible, then σuh suppµ UH ; if H is not invertible, then σuh B0, H 4 Some Notation The following notation will be used in the rest of the paper M, τ M C, 1Tr M C, 1 Tr denotes the reduced free product von Neumann algebra of M C with M C with the unique tracial state τ; M C 1 : M C, 1 Tr 1 and M C : 1 M C, 1 Tr; {E ij } i,j1,, {F ij } i,j1, are matrix units of M C 1 and M C, respectively; P E 11 and Q F 11 ; M N M C 1 x1 x N M C For X M, X x 3 x 4 1 x 1 x x 3 x means the decomposition is with respect to above matrix units of 4 M C 1 and M C, respectively W 0, W 0 1 1, W 0 1, W V , V , V , V A, A 1,, A n denote elements in M C 1, B, B 1,, B n denote elements in M C, X, Y, Z denote general elements in M; An element X in M is called centered if τx 0 We end this section with the following lemma The proof is an easy exercise Lemma V 1 M C 1 V 1 is free with M C 1 ; ; 1

6 3 Spectra of elements in the universal free product of M C and M C Let Å M C M C denote the universal free product C*-algebra of M C with M C Then there is a * homomorphism π from Å onto the reduced free product C*-algebra of M C and M C, the C*-subalgebra generated by M C 1 and M C in M Since σπa σa for a Å, it is useful to obtain some information of spectrum of AB, where A M C 1 and B 1 M C 31 Free product of normal matrices Lemma 31 Let A M C 1 and B 1 M C be normal matrices Then rab A B relative to Å Proof rab AB A B We need only to prove rab A B Since A is a normal matrix, there is a unitary matrix U 1 M C 1 such that U 1 AU1 α1 0 and α 0 β 1 A Similarly, there is a unitary matrix U 1 M C such 1 that U BU α 0 and α 0 β B Let π 1 X U 1 XU1 and π Y U Y U be -representations of M 1 C 1 and 1 M C to M C, respectively Then there is a α1 α -representation π π 1 π from Å to M C and πab 0 Therefore, 0 β 1 β α 1 α σπab σab So rab α 1 α A B Corollary 3 Let A M C 1 and B M C be normal matrices Then rab A B relative to M Proof We may assume that A and B are diagonal matrices Then we can treat AB as an operator in the full free product C Z Z Same technique used in the previous lemma gives the corollary 3 Free product of non-normal matrices It is well-known that two matrices X, Y in M C are unitarily equivalent if and only if TrX TrY, TrX TrY and TrX X TrY Y The proof of the following lemma now is an easy exercise Lemma 33 If A M C and TrA 0, then A is unitarily equivalent to a matrix of 0 α form, where α, β are complex numbers β 0 6

7 7 Remark 34 We have the following useful observations: α β 1 0 β α α e iθ e iθ 1 θ / β e iθ 0 0 e iθ 1 θ / e iθ 1+θ / 0 α β 0 Lemma 35 Let A M C 1 and B 1 M C be matrices such that TrA TrB 0 Then rab A B relative to Å Proof We need only to prove rab A B By Lemma 33 and Remark 34, there are 0 unitary matrices U, V in M C such that UAU α1 0 and V BV β 1 0 α β 0 and α 1 A, β B Let π 1 X UXU and π Y V Y V be -representations of M C 1 and 1 M C to M C, respectively Let π π 1 π be the induced - representation of Å to M C Then σab σπab σπ 1 Aπ B {α 1 β, α β 1 } Therefore, rab α 1 β A B Theorem 36 Let A M C 1 and B 1 M C be matrices such that TrA TrB 0 Then σab [ A 1 1 B 1 1, A B ] p [0, π], where p denotes the polar set product {re iθ : r [ A 1 1 B 1 1, A B ], θ [0, π]} Proof We will prove the theorem for two cases Case 1 Either A or B is not invertible We may assume that A is not invertible By 0 α1 TrA 0, Lemma 33 and Remark 34, A is unitarily equivalent to Without loss of generality, we assume that A M 0 0 C 1 By Lemma 33 and Remark 0 α 34, we may also assume that B 1 M β 0 C and β α 0 We need to prove that σab is the closed disc of complex plane with center 0 and radius β Since A is unitarily equivalent to e iθ A in M C 1, σab is rotation invariant For θ [0, π], cos θ sin θ let U Let π sin θ cos θ 1 X X and π Y UY U be -representations of M C 1 and 1 M C to M C, respectively Let π π 1 π be the induced -representation of Å to M C Then πab AUBU α sin θ + β cos θ α + β sin θ cosθ 0 0

8 8 So σπab { α sin θ + β cos θ, 0} Since [0, β] [ α, β] { α sin θ + β cos θ : θ [0, π]}, [0, β] σab Since σab is rotation invariant, σab contains the closed disc with center 0 and radius β By Lemma 35, σab is the closed disc of complex plane with center 0 and radius β Case Both A and B are invertible By Lemma 33 and Lemma 34, we may assume that β A and B such that β β 1 0 β 0 1, β 1 Then A and β 1 B 1 We need to prove that σab [1, β β ] p [0, π] By Lemma 35, rab β 1 β and rab 1 1 This implies that σab [1, β 1 β ] p [0, π] So we need only to prove σab [1, β 1 β ] p [0, π] cosψ e For φ, ψ [0, π], let U iφ sin ψ sin ψ e iφ Then U is a unitary matrix Let cosψ π 1 X UXU and π Y Y be -representations of M C 1 and 1 M C to M C, respectively Let π π 1 π be the induced -representation of Å to M C Then β1 β πab e iφ sin ψ + β e iφ cos ψ β 1 e iφ cos ψ e iφ sin ψ Let λ 1 φ, ψ, λ φ, ψ be the eigenvalues of πab Then λ 1 φ, ψλ φ, ψ detπab deta detb β 1 β, 31 λ 1 φ, ψ + λ φ, ψ β 1 e iφ + β e iφ cos ψ β 1 β e iφ + e iφ sin ψ 3 Note that σab {λ 1 φ, ψ : φ, ψ [0, π]} We only need to prove that {λ 1 φ, ψ : φ, ψ [0, π]} [1, β 1 β ] p [0, π] For this purpose, we need to show for any r [1, β 1 β ], θ [0, π], there are φ, ψ [0, π] such that re iθ + β 1β r e iθ β 1 e iφ + β e iφ cos ψ β 1 β e iφ + e iφ sin ψ 33 Let α cos ψ Simple computations show that equation 33 is equivalent to the following r + β 1β cosθ + i r β 1β sin θ r r Let α1 + β β 1 + β 1 β cosφ + iαβ 1 1β β 1 β sin φ Ω 1 { r + β 1β cos θ + i r β } 1β sin θ : r [1, β 1 β ], θ [0, π], r r

9 9 Ω {α1 + β β 1 + β 1 β cosφ + iαβ 1 1β β 1 β sin φ : α [0, 1], φ [0, π]} Now we need only to prove Ω 1 Ω Note that Ω 1 is the union of a family of ellipses with center the origin point and semimajor axis and semiminor axis r + β 1β and r β 1β, r r 1 r β 1 β, respectively Similarly, Ω is the union of a family of ellipses with center the origin point and semimajor axis and semiminor axis α1 + β β 1 + β 1 β and αβ 1 1β β 1 β, 0 α 1, respectively Note that the largest ellipse in Ω 1 is with semimajor axis and semiminor axis 1 + β 1 β and β 1 β 1, respectively; the smallest ellipse in Ω 1 is with semimajor axis and semiminor axis β 1 β and 0, respectively The largest ellipse in Ω is with semimajor axis and semiminor axis 1 + β 1 β and β 1 β 1, respectively; the smallest ellipse in Ω is with semimajor axis and semiminor axis 0 and β 1β 1 β 1 So both Ω +1 1 and Ω are the closure of the domain enclosed by the ellipse with center the origin point and semimajor axis and semiminor axis 1 + β 1 β and β 1 β 1, respectively Thus Ω 1 Ω 4 R-diagonal operators in M In this section, we prove the following result We will use the notation introduced in section 4 Theorem 41 In M, let A M C 1 and B M C Then AB is an R-diagonal operator if and only if τa τb 0 To prove Theorem 41, we need the following lemmas Lemma 4 {W 1, V 1, W 3 V 3 } LZ LZ LZ Proof Let U W 3 V 3 Then U is a Haar unitary operator We need only to prove that U is * free with the von Neumann subalgebra generated by W 1 and V 1 Let g 1 g g n be an alternating product of {U n : n 0} and {W 1, V 1, W 1 V 1, V 1 W 1, W 1 V 1 W 1, V 1 W 1 V 1, } By regrouping, it is an alternating product of {W 1, W 1 W 3, W3 W 1, W3 W 1W 3, W 3, W3 } and {V 1, V 3 V 1, V 1 V3, V 3 V 1 V3, V 3, V3 } Thus the trace is 0 0 α1 0 α Lemma 43 is an R-diagonal operator β 1 0 β 0 Proof Note that 0 α1 0 α β 1 0 β α1 0 0 β β 0 0 α By Lemma 4 and basic properties of R-diagonal operators given in, we prove the lemma

10 10 Lemma 44 With the assumption of Theorem 41 and assume AB is an R-diagonal operator and τa 0 Then τb 0 Proof Since AB is an R diagonal operator, τab 0 Since A, B are -free, τaτb τab 0 If τb 0, then done Otherwise, assume τa 0 Then 0 τabab τa BτB τa τb By assumption, τb 0 Lemma 45 Let B M C and λ be any complex number Then σe 1 B σe 1 λ+ B Proof By Jacobson s theorem, σe 1 λ + B {0} σe 11 E 1 λ + B {0} σe 1 λ + BE 11 {0} σe 1 BE 11 {0} σbe 1 {0} Proof of Theorem 41 If τa τb 0, then by Lemma 33 and Lemma 43, AB is an R-diagonal operator Conversely, assume that AB is an R-diagonal operator Then 0 τab τa τb So either τa 0 or τb 0 Without loss of generality, we assume that τa 0 If τa 0, then τb 0 by Lemma 44 If τa 0, then A is unitary equivalent to αe 1 We may assume that A E 1 By Theorem 1, if E 1 B is an R-diagonal operator, then re 1 B τb E 1 E 1 B τe 1 E 1 BB E 1 B Note that E 1B τb is an R-diagonal operator, re 1 B τb E 1 B τb By Lemma 45, B B τb This implies that τb 0 This ends the proof Combining Theorem 41, Theorem 1 and transform of Voiculescu see [16, 17], we have the following theorem It is interesting to compare the following theorem and Theorem 36 Theorem 46 Let A M C 1, B M C and τa τb 0 Then 1 µ AB is rotation invariant; σab suppµ AB [ A 1 1 B 1 1, A B ] p [0, π]; 3 µ AB {0} max{µ A A{0}, µ B B{0}} and µ AB B0, µa A µb B t 1 1/ t, for t [µ AB {0}, 1]

11 11 5 Algebraic techniques For X M, define E11 XE ΦX 11 E 11 XE 1 E 1 XE 11 E 1 XE 1 Then Φ is a -isomorphism from M onto E 11 ME 11 M C 1 We will identify M with E 11 ME 11 M C 1 by the canonical isomorphism Φ In [4], K Dykema proved that E 11 ME 11 LF3 For B M C, we may write b11 b B 1 b 1 b with respect to matrix units in M 1 Then b ij LF 3 In this section, we will develop the algebraic techniques used in [4] Combining the matrix techniques, we may explicitly express b ij in terms of free generators of LF 3 Let Λ{W 1, V 1 } be the set of words generated by W 1, V 1 Note that W 1 V 1 1 and τw 1 τv 1 0 The following observation is crucial in [4] The proof is an easy exercise Lemma 51 τg 1 g g n 0 for an alternating product of Λ{W 1, V 1 } \ {1, W 1 } and {E 1, E 1 } Recall that P E 11 and Q F 11 Let W be the polar part of 1 PQP and U E 1 W The following corollary is a special case of Theorem 35 of [4] Corollary 5 U is a Haar unitary operator in M P P MP and U, PQP are -free in M P With the canonical identification of M with M P M C 1, Q PQP PQP PQP U U PQP PQP U 1 PQPU By [16], the distribution of PQP relative to M P is non-atomic and the density function is ρt 1 1, 0 t 1 51 π t By Corollary 5, the von Neumann subalgebra M 1 generated by M C 1 and Q is -isomorphic to LF M C 1 Since M 1 is also -isomorphic to M C LZ,

12 1 M C LZ LF M C, which is proved by Dykema in [4] Since V 1 Q 1, V 1 PQP 1 U PQP PQP PQP PQP U U 1 PQPU Simple computation shows that the density function of PQP 1 is σt 1 π 1, 1 t 1 1 t Let H PQP 1, then V 1 H U 1 H 1 H U U HU Let H V H be the polar decomposition of H Since H is a symmetric selfadjoint operator, V 1 and V is independent with the von Neumann algebra generated by H in the classical probability sense Let h H, u V U, v UV Then u, v are Haar unitary operators and the distribution of h relative to M P is non-atomic Lemma 53 h, u, v are * free Proof Let g 1 g g n be an alternating product of elements of S { H } CI, {V U n : n 0}, {UV n : n 0} By regrouping, it is an alternating product of elements of {S, V, V S, SV, V SV } and {U n : n 0} Since H and U are -free, {S, V, V S, SV, V SV } and {U n : n 0} are free Since V and S are independent, τv S τsv 0 for S S This implies that τg 1 g g n 0 By simple computations, we have the following h V 1 E 11 V 1 1 h hu u h 1 h u 1 h u, 5 HU V 1 E 1 V 1 1 H HU HU U 1 H U 1 H U 1 H U HU 53 hv 1 h hv hu u 1 h v 1 h u 1 h v hu 54

13 13 By Lemma, M M C 1 V 1 M C 1 V 1 M P M C 1 With this isomorphism, M P is the von Neumann algebra generated by h, u and v by 5 and 54 So M P LF3 By simple computations, we have V 1 α β γ σ 1 b11 b V 1 1 b 1 b where b 11 σ + α σh + γ 1 h vh + βhv 1 h, b 1 α σh 1 h u + γ 1 h v 1 h u βhv hu, b 1 α σu h 1 h γu hvh + βu 1 h v 1 h, b α + σ αu h u γu hv 1 h u βu 1 h v hu Theorem 54 M α β LF 3 M C 1 ; furthermore, let B γ σ b11 b then with respect to the matrix units {E ij } i,j1, M C 1, B 1 b 1 b b ij are given as above, in M C,, where 1 Example 55 In Theorem 54, let β γ 0 Then we have α 0 σ + α σh α σh 1 h u 0 σ α σu h 1 h α + σ αu h u 1 Example 56 In Theorem 54, let α σ and γ 0 Then we have α β α + βhv 1 h βhv hu 0 α βu 1 h v 1 h α βu 1 h v hu 1 Remark 57 By equation 5, the distribution of h is the distribution of E 11 V 1 E 11 V 1 E 11 relative to M P So the distribution of h is same as the distribution of PQP relative to M P By [Vo], the distribution of PQP relative to M P is non-atomic and the density function is ρt 1 1, 0 t 1 π t

14 6 Miscellaneous examples 0 1 Example 61 We compute the Brown spectrum of αe 1 +βf 1 Let F b1 b Then b 3 b 4 1 αe 1 + βf 1 αβe 1 F 1 + F 1 E 1 αβe 1 + F 1 b3 b αβ 1 + b 4 0 b 3 So µ αe1 +βf 1 µ αβb 3 By equation 53, the distribution of b 3 is same as the distribution of U 1 H Since U 1 H is an R-diagonal operator, U 1 H is also an R- diagonal operator Since the distribution of αe 1 +βf 1 is rotation invariant, µ αe1 +βf 1 µ αβ b, where b U 1 H Simple computations show that or by Proposition 510 and Corollary 511 of [5] Hence dµ b z 1 π dµ αe1 +βf 1 z dµ αβ b z 1 π 1 1 r drdθ 0 r 1 αβ 3/ αβ r drdθ and αβ σαe 1 + βf 1 B 0, αβ Corollary 6 rαe 1 + βf 1 0 r αβ Corollary 63 Let A M C 1 and B M C Then A + B is an R-diagonal operator if and only if A + B 0 Proof Indeed, if A + B is an R-diagonal operator, then τa + B 0 So we may assume that τa τb 0 Let λ, λ and η, η be the spectra of A and B, respectively Then 0 τa + B τa + τb λ + η By simple computation we have 0 τa+b 4 τa 4 +τb 4 +4τA τb λ 4 +η 4 +4λ η λ +η +λ η λ η Thus λ η 0 This implies that A and B are unitary equivalent to αe 1 and βf 1, αβ respectively By Corollary 6, ra+ B On the other hand, since we assume that A + B is an R-diagonal operator, by Theorem 1, ra + B τa + B A + B τᾱe 1 + βf 1 αe 1 + βf 1 α + β So α + β αβ This implies that α 0 and β 0 Hence A + B

15 15 Example 64 We compute the spectrum and Brown spectrum of 1 0 α β X α By Example 56, we have the following 1 0 α β 1 0 X α α + βhv 1 h βhv hu 0 0 α + βhv 1 h βhv hu βu 1 h v 1 h α βu 1 h v hu So σx {0} σα+βhv 1 h and µ X 1 δ 0+ 1 µ α+βhv 1 h Note that µ hv 1 h µ v 1 h h and v 1 h h is an R-diagonal operator We have the following computations: h h τ P1 h h τ P 1 PQPPQP By Theorem 1, π t1 tdt t 1 h h 1 τ P 1 h h 1 τ P 1 PQPPQP π dt t1 t t σx suppµ X {0} Bα, β / Example 65 We compute the spectrum and Brown spectrum of 0 1 α 0 Y β By Example 55, we have the following 0 1 α Y β α βu h 1 h α + β αu h u , β + α βh α βh 1 h u α βu h 1 h α + β αu h u Since u h 1 h is an R-diagonal operator, similar computations as Example 64, we have σy suppµ Y B0, α β / 1 1

16 16 Example 66 We compute the spectrum and Brown spectrum of 1 α 1 β Z 1 + αe βf For λ C, we have Z λ1 1+αE 1 1+βF 1 λ1+αe 1 1 αe 1 1+αE 1 λαe 1 +βf 1 λ 1 This implies that λ σz if and only if λ 1 σλαe 1 + βf 1 By Example 61, λ 1 σλαe 1 + βf 1 if and only if λ 1 αβ λ So { σz λ C : λ 1 αβ λ } { } In the following, we will show that suppµ Z σz λ C : λ 1 αβ λ For { } this purpose, we need only to prove that suppµ Z 1 σz 1 λ C : λ αβ λ+1 1 Note that 1 + αe 1 1 For λ C, we have log Z 1 λ log 1 + αe βf λ1 + αe 1 1 αe 1 log 1 + αe 1 + log 1 + βf 1 λ λαe 1 log 1 + λαe 1 + βf 1 λ By Example 61, µ 1+λαE1 +βf 1 µ 1+λ αβ b Hence, log Z 1 λ log 1 + λ αβ b λ λ λ log b log + log λ 1 + λ αβ 1 + λ αβ Since b is an R-diagonal operator, this implies that λ λ log b log log λ + log Z 1 λ λ αβ 1 + λ αβ Suppose λ 0 σz 1 and λ 0 / suppµ Z 1 Then there is δ > 0 such that Bλ 0, δ C \ suppµ Z 1 Now log Z 1 λ is a harmonic function on Bλ 0, δ Since τz 1 n 0

17 17 for all n 1,, By Lemma 43 of [8], for λ C such that λ rz 1, log Z 1 λ log λ By the uniqueness of harmonic functions, we have log Z 1 λ log λ for λ Bλ 0, δ By equation 61, this implies that log b λ λ log λ αβ 1 + λ αβ Let r λ Then equation 6 implies that 1+λ αβ log b r log r for r s, t [0, 1 ] Since b is an R-diagonal operator, this implies that log b z is harmonic on the annulus with inner radius s and outer radius t, 0 < s < t < 1 By Theorem 1, suppµ b B0, 1 It is a contradiction 7 Hyperinvariant subspaces for operators in M Lemma 71 For X M, if suppµ X {λ}, then τx n λ n for n 1,, Proof τx n suppµ X z n dµ X z λ n The converse of Lemma 71 is not true Since for an R-diagonal operator X, we have τx n 0 for n 1, Proposition 7 Let X A + B, where A M C 1 and B M C If A, B are not scalar matrices, then suppµ X contains more than two points Proof Suppose A, B are not scalar matrices Since A + B τa1 + τb1 + A τa1 + B τb1, to show suppµ X contains more than two points, we need only to show µ A τa1+b τb1 contains more than two points So we may assume that τa τb 0 and A, B 0 Assume that the spectra of A and B are λ 1, λ 1 and λ, λ If τa τb 0, then A and B are unitarily equivalent to αe 1 and βf 1 in M C 1 and M C, respectively Thus µ X µ αe1 +βf 1 By Example 61, suppµ X contains more than two points Now suppose τa 0 or τb 0 Without loss of generality, we assume that λ 1 τa 0 Note that τa+b 0 and τa+b τa +τb λ 1 + λ If λ 1 + λ 0, by Lemma 71, suppµ X contains more than two points Suppose λ 1 + λ 0 Then τb λ 0 Simple computations show that τa+b 4 τa 4 +τb 4 +4τA τb λ 4 1+λ 4 +4λ 1λ λ 1+λ +λ 1λ λ 1λ 0 Note that τa + B 0 By Lemma 71, suppµ X contains more than two points

18 18 Proposition 73 Let X AB, where A M C 1 and B M C If A, B are not scalar matrices, then suppµ X contains more than two points Proof Suppose A, B are not scalar matrices We consider the following cases: Case 1 τa τb 0 and A, B 0 By Theorem 41, AB 0 is an R-diagonal operator So suppµ X contains more than two points Case τa 0, τb 0 or τa 0, τb 0 Without loss of generality, we assume that τa 0 and τb 0 Then τab 0 and τabab τa τb If τa 0, then τabab 0 By Lemma 101, suppµ X contains more than two points If τa 0, then A is unitarily equivalent to αe 1 in M C 1 By Lemma 45, µ X µ αe1 B τb Since αe 1 B τb 0 is an R-diagonal operator, suppµ X contains more than two points Case 3 τa 0 and τb 0 We may assume that τa τb 1 Let A 1 + A 1 and B 1 + B 1 Then τa 1 τb 1 0 Subcase 31 τa 1 0 or τb 1 0 We may assume that τa 1 0 Simple computation shows that τab 1, τabab 1+τA 1 +τb 1 and τab3 1+3τA 1 + τb1 + 9τA 1τB1 If τa 1 + τa 0, then τabab 1 By Lemma 71, suppµ X contains more than two points If τa 1 + τa 0, then τa τa 1 0 So τab 3 1 By Lemma 71 again, suppµ X contains more than two points Subcase 3 τa 1 τa 0 Then A 1 and A are unitarily equivalent to αe 1 and βf 1 in M C 1 and M C, respectively So µ X µ 1+αE1 1+βF 1 We may assume 1 α 1 β that A and B By Example 66, suppµ X contains more than 1 two points Corollary 74 Let X AB or X A + B, where A M C 1 and B M C If X λ1, then X has a nontrivial hyperinvariant subspace relative to M Proof If X A+B and A λ1 or B λ1, then X λ1+b or X λ1+a If X is not a scalar matrix and η is an eigenvalue of X, then kerx η1 is a nontrivial hyperinvariant subspace of X If X A + B and A, B λ1, then suppµ X contains more than two points by Proposition 73 By [9], X has a nontrivial hyperinvariant subspace relative to M If X AB and A λ1 or B λ1, then X λb or X λa If X is not a scalar matrix and η is an eigenvalue of X, then kerx η1 is a nontrivial hyperinvariant subspace of X If X AB and A, B λ1, then suppµ X contains more than two points By [9], X has a nontrivial hyperinvariant subspace relative to M

19 19 Acknowledgements: The authors want to express their deep gratitude to professor Eric Nordgren for valuable discussions The authors also thank the referee for some useful suggestions References [1] P Biane, FLehner, Computation of some examples of Brown s spectral measure in free probability, Colloq Math , [] LG Brown, Lidskii s theorem in the type II case, Geometric methods in operator algebras, H Araki and E Effros Eds Pitman Res notes in Math Ser 13, Longman Sci Tech 1986, 1-35 [3] WM Ching, Free products of von Neumann algebras, Trans Amer Math Soc 178, 1973, [4] K Dykema, Interpolated free group factors, Pacific J Math 163, 1994, [5] J Fang, D Hadwin, R Mohan, On transitive algebras containing a standard finite von Neumann subalgebra, to appear on JFunctAnal [6] B Fuglede and RVKadsion, Determinant theory in finite factors, Annals of Math bf , [7] FP Greenleaf, Invariant means on topological groups and their applications Van Nostrand Mathematical Studies, No 16 Van Nostrand Reinhold Co, New York-Toronto, Ont-London 1969 [8] U Haagerup and F Larsen, Brown s spectral distribution measure for R diagonal elements in finite von Neumann algebras, JFunctAnal , [9] U Haagerup and H Schultz, Invariant Subspaces for Operators in a General II 1 -factor, preprint available at [10] U Haagerup and H Schultz, Brown measures of unbounded operators affiliated with a finite von Neumann algebra, To appear in Math Scand, preprint available at [11] F Larsen, Powers of R-diagonal elements J Operator Theory 47 00, no 1, [1] A Nica and R Speicher, R diagonal pairs a common approach to Haar unitaries and circular elements Fields Institute Communications, 1997,

20 0 [13] A Nica and R Speicher, Commutators of free random variables Duke Math J 93, 1998, [14] H Radjavi and P Rosenthal, Invariant Subspaces, Springer-Verlag, New York, 1973 [15] H Schultz, Brown measures of sets of commuting operators in a II 1 -factor J Funct Analysis , [16] DV Voiculescu, Multiplication of certain noncommuting random variables J Operator Theory 18, 1987, no, 3 35 [17] DV Voiculescu, K Dykema and A Nica, Free Random Variables, CRM Monograph Series, vol 1, AMS, Providence, RI, 199 address: [Junsheng Fang] jfang@cisunix unhedu address: [Don Hadwin] don@mathunhedu address: [Xiujuan Ma] mxjsusan@@hebuteducn