Principal Angles Between Subspaces and Their Tangents

Transcription

1 MITSUBISI ELECTRIC RESEARC LABORATORIES Principal Angles Between Subspaces and Their Tangents Knyazev, AV; Zhu, P TR September 2012 Abstract Principal angles between subspaces (PABS) (also called canonical angles) serve as a classical tool in mathematics, statistics, and applications, eg, data mining Traditionally, PABS are introduced and used via their cosines The tangents of PABS have attracted relatively less attention, but are important for analysis of convergence of subspace iterations for eigenvalue problems We explicitly construct matrices, such that their singular values are equal to the tangents of PABS, using several approaches: orthonormal and nonorthonormal bases for subspaces, and orthogonal projectors Cornell University Library This work may not be copied or reproduced in whole or in part for any commercial purpose Permission to copy in whole or in part without payment of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of Mitsubishi Electric Research Laboratories, Inc; an acknowledgment of the authors and individual contributions to the work; and all applicable portions of the copyright notice Copying, reproduction, or republishing for any other purpose shall require a license with payment of fee to Mitsubishi Electric Research Laboratories, Inc All rights reserved Copyright c Mitsubishi Electric Research Laboratories, Inc, Broadway, Cambridge, Massachusetts 02139

2 MERLCoverPageSide2

3 Principal angles between subspaces and their tangents Peizhen Zhu a, Andrew V Knyazev a,b, arxiv: v1 [mathna] 4 Sep 2012 a Department of Mathematical and Statistical Sciences; University of Colorado Denver, PO Box , Campus Box 170, Denver, CO , USA b Mitsubishi Electric Research Laboratories; 201 Broadway Cambridge, MA Abstract Principal angles between subspaces (PABS) (also called canonical angles) serve as a classical tool in mathematics, statistics, and applications, eg, data mining Traditionally, PABS are introduced and used via their cosines The tangents of PABS have attracted relatively less attention, but are important for analysis of convergence of subspace iterations for eigenvalue problems We explicitly construct matrices, such that their singular values are equal to the tangents of PABS, using several approaches: orthonormal and nonorthonormal bases for subspaces, and orthogonal projectors Keywords: principal angles, canonical angles, singular value, projector 2000 MSC: 15A42, 15A60, 65F35 1 Introduction The concept of principal angles between subspaces (PABS) is first introduced by Jordan [1] in 1875 otelling [2] defines PABS in the form of canonical correlations in statistics in 1936 Numerous researchers work on PABS; see, eg, our pseudo-random choice of initial references [3, 4, 5, 6, 7, 8], out of tens of thousand of Internet links for the keywords principal angles and canonical angles September 5, 2012 The preliminary version is at This material is based upon work partially supported by the National Science Foundation under Grant No Corresponding author addresses: peizhenzhu[at]ucdenveredu (Peizhen Zhu), andrewknyazev[at]ucdenveredu (Andrew V Knyazev) URL: (Andrew V Knyazev) Preprint submitted to Linear Algebra and Its Applications September 5, 2012

4 In this paper, we first briefly review the concept and some important properties of PABS in Section 2 Traditionally, PABS are introduced and used via their sines and more commonly, because of the connection to canonical correlations, cosines The tangents of PABS have attracted relatively less attention, despite of the celebrated work of Davis and Kahan [9], which includes several tangent-related theorems Our interest to the tangents of PABS is motivated by their applications in theoretical analysis of convergence of subspace iterations for eigenvalue problems We review some previous work on the tangents of PABS in Section 3 The main goal of this work is explicitly constructing a family of matrices such that their singular values are equal to the tangents of PABS We form these matrices using several different approaches: orthonormal bases for subspaces in Section 4, non-orthonormal bases in Section 5, and orthogonal projectors in Section 6 2 Definition and basic properties of PABS In this section, we remind the reader the concept of PABS and some fundamental properties of PABS First, we recall that an acute angle between two unit vectors x and y, ie, with x x = y y = 1, is defined as cos θ(x, y) = x y, where 0 θ(x, y) π/2 The definition of an acute angle between two vectors can be recursively extended to PABS; see, eg, [2, 10, 11] Definition 21 Let X C n and Y C n be subspaces with dim(x ) = p and dim(y) = q Let m = min (p, q) The principal angles Θ(X, Y) = [θ 1,, θ m ], where θ k [0, π/2], k = 1,, m, between X and Y are recursively defined by subject to s k = cos(θ k ) = max x X max y Y x y = x k y k, x = y = 1, x x i = 0, y y i = 0, i = 1,, k 1 The vectors {x 1,, x m } and {y 1,, y m } are called the principal vectors An alternative definition of PABS, from [10, 11], is based on the singular value decomposition (SVD) and reproduced here as the following theorem 2

5 Theorem 21 Let the columns of matrices X C n p and Y C n q form orthonormal bases for the subspaces X and Y, correspondingly Let the SVD of X Y be UΣV, where U and V are unitary matrices and Σ is a p q diagonal matrix with the real diagonal elements s 1 (X Y ),, s m (X Y ) in nonincreasing order with m = min(p, q) Then cos Θ (X, Y) = S ( X Y ) = [ s 1 ( X Y ),, s m ( X Y )], where Θ (X, Y) denotes the vector of principal angles between X and Y arranged in nondecreasing order and S(A) denotes the vector of singular values of A Moreover, the principal vectors associated with this pair of subspaces are given by the first m columns of XU and Y V, correspondingly Theorem 21 implies that PABS are symmetric, ie Θ(X, Y) = Θ(Y, X ), and unitarily invariant, ie, Θ(UX, UY) = Θ(X, Y) for any unitary matrix U C n n, since (UX) (UY ) = X Y A number of other important properties of PABS have been established, for finite dimensional subspaces, eg, in [4, 5, 6, 7, 8], and for infinite dimensional subspaces in [3, 12] We list a few most useful properties of PABS below Property 21 [7] In the notation of Theorem 21, let p q and let [X, X ] be a unitary matrix such that S(X Y ) = [s 1(X Y ),, s q(x Y )] where s 1 (X Y ) s q(x Y ) Then s k(x Y ) = sin(θ q+1 k), k = 1,, q Relationships of principal angles between X and Y, and between their orthogonal complements are thoroughly investigated in [4, 5, 12] The nonzero principal angles between X and Y are the same as those between X and Y Similarly, the nonzero principal angles between X and Y are the same as those between X and Y Property 22 [4, 5, 12] Let the orthogonal complements of the subspaces X and Y be denoted by X and Y, correspondingly Then, 1 [ Θ (X, Y), 0,, 0 ] = [ Θ (X, Y ), 0,, 0 ] 2 [ Θ (X, Y ), 0,, 0 ] = [ Θ (X, Y), 0,, 0 ] 3 [ π,, π, 2 2 Θ (X, Y) ] = [ π 2 Θ (X, Y ), 0,, 0 ], where there are max(dim(x ) dim(y), 0) additional π/2s on the left Extra 0s at the end may need to be added on either side to match the sizes Symbols and denote the components in vector arranged in nonincreasing and nondecreasing order, correspondingly These statements are widely used to discover new properties of PABS 3

6 3 Tangents of PABS The tangents of PABS serve as an important tool in numerical matrix analysis For example, in [13, 14], the authors use the tangent of the largest principal angle derived from a norm of a specific matrix In [6, p ] and [15, Theorem 24, p 252] the tangents of PABS, related to singular values of a matrix without an explicit matrix formulation are used to analyze perturbations of invariant subspaces In [6, 13, 14, 15], the two subspaces have the same dimensions Let the orthonormal columns of matrices X, X, and Y span the subspaces X, the orthogonal complement X of X, and Y, correspondingly According to Theorem 21 and Property 21, cos Θ(X, Y) = S(X Y ) and sin Θ(X, Y) = S(X Y ) The properties of sines and especially cosines of PABS are well studied; eg, in [7, 10] One could obtain the tangents of PABS by using the sines and the cosines of PABS directly, however, in some situations this does not reveal enough information about their properties In this work, we construct a family F of explicitly given matrices, such that the singular values of the matrix T are the tangents of PABS, where T F We form T in three different ways First, we derive T as the product of matrices whose columns form the orthonormal bases of subspaces Second, we present T as the product of matrices with non-orthonormal columns Third, we form T as the product of orthogonal projections on subspaces To better understand the matrix T, we provide a geometric interpretation of singular values of T and of the action of T as a linear operator Our constructions include the matrices used in [6, 13, 14, 15] Furthermore, we consider the tangents of principal angles between two subspaces not only with the same dimensions, but also with different dimensions 4 tan Θ in terms of the orthonormal bases of subspaces The sines and cosines of PABS are obtained from the singular values of explicitly given matrices, which motivates us to construct T as follows: if matrices X and Y have orthonormal columns and X Y has full rank, let T = X Y ( X Y ) 1, then tan Θ(X, Y) could be equal to the positive singular values of T We begin with an example in two dimensions in Figure 1 Let X = ( cos β sin β ), X = ( sin β cos β ), and Y = ( cos (θ + β) sin (θ + β) ), 4

7 Figure 1: The PABS in 2D where 0 θ < π/2 and β [0, π/2] Then, X Y = cos θ and X Y = sin θ Obviously, tan θ coincides with the singular value of T = X Y ( X Y ) 1 If θ = π/2, then the matrix X Y is singular, which demonstrates that, in general, we need to use its Moore-Penrose Pseudoinverse to form our matrix T = X Y ( X Y ) The Moore-Penrose Pseudoinverse is well known For a matrix A C n m, its Moore-Penrose Pseudoinverse is the unique matrix A C m n satisfying AA A = A, A AA = A, (AA ) = AA, (A A) = A A Some properties of the Moore-Penrose Pseudoinverse are listed as follows If A = UΣV is the SVD of A, then A = V Σ U If A has full column rank and B has full row rank, then (AB) = B A owever, this formula does not hold in general AA is the orthogonal projector onto the range of A, and A A is the orthogonal projector onto the range of A For additional properties of the Moore-Penrose Pseudoinverse, we refer the reader to [6] Now we are ready to prove that the intuition discussed above, suggesting to try T = X Y ( X Y ), is correct Moreover, we cover the case of the subspaces X and Y having possibly different dimensions 5

8 Theorem 41 Let X C n p have orthonormal columns and be arbitrarily completed to a unitary matrix [X, X ] Let the matrix Y C n q be such that Y Y = I Then the positive singular values, denoted by S + (T ), of the matrix T = X Y ( X Y ) satisfy tan Θ(R(X), R(Y )) = [,,, S + (T ), 0,, 0], where R( ) denotes the matrix range Proof Let [ Y Y ] be unitary, then [ X X ] [ Y Y ] is unitary, such that [ X X ] [ Y Y ] = p n p [ q n q X Y X Y X Y X Y ] Applying the CS-decomposition (CSD), eg, [16, 17, 18], we get [ ] X [ X X ] [ Y Y ] = Y X Y X Y = X Y [ U1 U 2 ] [ V1 D V 2 ], where U 1 U(p), U 2 U(n p), V 1 U(q), and V 2 U(n q) The symbol U(k) denotes the family of unitary matrices of the size k The matrix D has the following structure: D = r s q r s n p q+r s p r s r I Os s C S p r s O c I n p q+r O s I s S C q r s I O c, where C = diag(cos(θ 1 ),, cos(θ s )), and S = diag(sin(θ 1 ),, sin(θ s )) with θ k (0, π/2) for k = 1,, s, which are the principal angles between the subspaces R(Y ) and R(X) The matrices C and S could be not present The matrices O s and O c are matrices of zeros and do not have to be square I denotes the identity matrix We may have different sizes of I in D 6

9 Therefore, T = X Y ( X Y ) = U2 = U 2 O s SC 1 O s S U1 I V 1 V 1 I C O c U 1 O c ence, S + (T ) = (tan(θ 1 ),, tan(θ s )), where 0 < θ 1 θ 2 θ s < π/2 From the matrix D, we obtain S ( X Y ) = S (diag(i, C, O c )) According to Theorem 21, Θ(R(X), R(Y )) = [0,, 0, θ 1,, θ s, π/2,, π/2], where θ k (0, π/2) for k = 1,, s, and there are min(q r s, p r s) additional π/2 s and r additional 0 s on the right-hand side, which completes the proof Remark 42 If more information is available about the subspaces X, Y and their orthogonal complements, we can obtain the exact sizes of block matrices in the matrix D Let us consider the decomposition of the space C n into an orthogonal sum of five subspaces as in [12, 19], C n = M 00 M 01 M 10 M 11 M, where M 00 = X Y, M 01 = X Y, M 10 = X Y, M 11 = X Y (X = R (X) and Y = R (Y )) Using Tables 1 and 2 in [12] we get dim(m 00 ) = r, dim(m 10 ) = q r s, dim(m 11 ) = n p q + r, dim(m 01 ) = p r s, where M = M X M X = M Y M Y with M X =X (M 00 M 01 ), M X =X (M 10 M 11 ), M Y =Y (M 00 M 10 ), M Y =Y (M 01 M 11 ), 7

10 and s = dim(m X ) = dim(m Y ) = dim(m X ) = dim(m Y ) Thus, we can represent D in the following block form dim(m 00 ) dim(m)/2 dim(m 10 ) dim(m 11 ) dim(m)/2 dim(m 01 ) dim(m 00 ) I Os dim(m)/2 C S dim(m 01 ) O c I dim(m 11 ) O s I dim(m)/2 S C dim(m 10 ) I O c In addition, it is possible to permute the first q columns or the last n q columns of D, or the first p rows or the last n p rows and to change the sign of any column or row to obtain the variants of the CSD Remark 43 From Remark 42, we see that some identity and zero matrices may be not present in the matrix D To obtain more details for D, we have to deal with D in several cases based on either p q or q p and either p+q n or p + q n Since the angles are symmetric such that Θ(X, Y) = Θ(Y, X ), it is enough to discuss one case of either p q or q p Let us assume q p which implies dim(m 01 ) dim(m 10 ) = p q Case 1: assuming p + q n, we have (also see [16, 20]), C 0 O S 0 O O (p q) q D = O O I p q S 0 O C 0 O O (n p q) q I n p q O O where C 0 = diag(cos(θ 1 ),, cos(θ q )) and S 0 = diag(sin(θ 1 ),, sin(θ q )) with θ k [0, π/2] for k = 1,, q Case 2 : assuming p + q n, we have D =, C 0 O (n p) (p+q n) S 0 O O (p+q n) (n p) I p+q n O O O (p q) (n p) O (p q) (p+q n) O I p q S 0 O (n p) (p+q n) C 0 O where C 0 = diag(cos(θ 1 ),, cos(θ n p )) and S 0 = diag(sin(θ 1 ),, sin(θ n p )) with θ k [0, π/2] for k = 1,, n p, 8

11 Due to the fact that the angles are symmetric, ie, Θ(X, Y) = Θ(Y, X ), the matrix T in Theorem 41 could be substituted with Y X(Y X) Moreover, the nonzero angles between the subspaces X and Y are the same as those between the subspaces X and Y ence, T can be presented as X Y (X Y ) and Y X (Y X ) Furthermore, for any matrix T, we have S(T ) = S(T ), which implies that all conjugate transposes of T are valid Let F denote a family of matrices, such that the singular values of the matrix in F are the tangents of PABS To sum up, any of the formulas for T in F in the first column of Table 1 can be used in Theorem 41 Table 1: Different matrices in F: matrix T F using orthonormal bases X Y ( X Y ) P X Y ( X Y ) Y X(Y X) P Y X(Y X) X Y (X Y ) P X Y (X Y ) Y X (Y X ) P Y X (Y X ) (Y X) Y X (Y X) Y P X (X Y ) X Y (X Y ) X P Y (Y X ) Y X (Y X ) Y P X (X Y ) X Y (X Y ) X P Y Using the fact that the singular values are invariant under unitary multiplications, we can also use P X Y (X Y ) for T in Theorem 41, where P X is an orthogonal projector onto the subspace X Thus, we can use T as in the second column in Table 1, where P X, P Y, and P Y denote the orthogonal projectors onto the subspace X, Y, and Y, correspondingly Remark 44 From the proof of Theorem 41, we immediately derive that X Y ( X Y ) = (Y X ) Y X, and Y X(Y X) = (X Y ) X Y Remark 45 Let X Y ( X Y ) = U2 Σ 1 U 1 and Y X(Y X) = V 2 Σ 2 V 1, be the SVDs, where U i and V i (i = 1, 2) are unitary matrices and Σ i (i = 1, 2) are diagonal matrices Let the diagonal elements of Σ 1 and Σ 2 be in the same order Then the principal vectors associated with the pair of subspaces X and Y are given by the columns of XU 1 and Y V 1 5 tan Θ in terms of the bases of subspaces In the previous section, we use matrices X and Y with orthonormal columns to formulate T It turns out that we can choose a non-orthonormal 9

12 basis for one of the subspaces X or Y Theorem 51 Let [X, X ] be a unitary matrix with X C n p Let Y C n q and rank (Y ) = rank ( X Y ) p Then the positive singular values of the matrix T = X Y ( X Y ) satisfy tan Θ(R(X), R(Y )) = [S + (T ), 0,, 0] Proof Let the rank of Y be r, thus r p Let the SVD of Y be UΣV, where U is an n n unitary matrix; Σ is an n q real diagonal matrix with diagonal entries s 1,, s r, 0,, 0 ordered by decreasing magnitude such that s 1 s 2 s r > 0, and V is a q q unitary matrix Since rank(y )=r q, we can compute a reduced SVD such that Y = U r Σ r Vr Only the r column vectors of U and the r row vectors of V, corresponding to nonzero singular values are used, which means that Σ r is an r r invertible diagonal matrix Based on the fact that the left singular vectors corresponding to the non-zero singular values of Y span the range of Y, we have tan Θ(R(X), R(Y )) = tan Θ(R(X), R(U r )) Since rank(x Y ) = r, we have rank(x U r Σ r ) = rank(x U r ) = r Let T 1 = X U r ( X U r ) According to Theorem 41, it follows that tan Θ(R(X), R(U r )) = [S + (T 1 ), 0,, 0] It is worth noting that the angles between R(X) and R(U r ) are in [0, π/2), since X U r is full rank Our task is now to show that T 1 = T By direct computation, we have T = X Y ( X Y ) = X U r Σ r V r ( X U r Σ r V = X U r Σ r Vr ( ) = X U r Σ r X U r Σ r = X U r Σ r Σ r r ) ( V r ) ( X U r Σ r ) ( X U r ) = X U r ( X U r ) In two identities above we use the fact that if a matrix A is of full column rank, and a matrix B is of full row rank, then (AB) = B A ence, tan Θ(R(X), R(Y )) = [S + (T ), 0,, 0] which completes the proof 10

13 Remark 52 Let X Y have full rank Let us define Z = X +X T following [6, p ] and [15, Theorem 24, p252] Since ( X Y ) X Y = I, the following identities Y = P X Y + P X Y = XX Y + X X Y = ZX Y imply that R(Y ) R(Z) By direct calculation, we obtain that X Z = X (X +X T ) = I and Z Z = (X +X T ) (X +X T ) = I +T T Thus, X Z ( Z Z ) 1/2 = (I + T T ) 1/2 is ermitian positive definite The matrix Z(Z Z) 1/2 by construction has orthonormal columns which span the space Z Moreover, we observe that S ( X Z(Z Z) 1/2) = Λ ( (I + T T ) 1/2) = ( 1 + S 2 (T ) ) 1/2 Therefore, tan Θ(R(X), R(Z)) = [S + (T ), 0,, 0] and dim(x ) = dim(z) From Theorem 51, we have tan Θ(R(X), R(Y )) tan Θ(R(X), R(Z)) It is worth noting that Θ(R(X), R(Y )) and Θ(R(X), R(Z)) in (0, π/2) are the same For the case p = q, we have that Θ(R(X), R(Y )) = Θ(R(X), R(Z)) This gives us an explicit matrix expression X Y ( X Y ) 1 for the matrix P in [6, p ] and [15, Theorem 24, p252] Remark 53 The condition rank (Y ) = rank ( X Y ) p is necessary in Theorem 51 For example, let X = 0, X = 1 0, and Y = Then, we have X Y = ( 1 1 ) and (X Y ) = ( 1/2 1/2 ) Thus, T = X Y ( X Y ) = ( 1/2 0 ), and s(t ) = 1/2 On the other hand, we obtain that tan Θ(R(X), R(Y )) = 0 From this example, we see that the result in Theorem 51 does not hold for the case rank(y ) = rank ( X Y ) > p Corollary 51 Using the notation of Theorem 51, let Y be full rank and p = q Let P X be an orthogonal ( projection onto the subspace R(X ) Then tan Θ(R(X), R(Y )) = S P X Y ( X Y ) ) 1 Proof Since the singular [ ( values are invariant under unitary transform, it is easy to obtain S X Y ( X Y ) ) ] ( 1, 0,, 0 = S P X Y ( X Y ) ) 1 Moreover, the number of the singular values of P X Y ( X Y ) 1 is p, hence we obtain the result as stated 11

14 Above we construct T explicitly in terms of bases of subspaces X and Y Next, we provide an alternative construction by adopting only the triangular matrix from the QR factorization of a basis of the subspace X + Y Corollary 52 Let X C n p and Y C n q be matrices of full rank where q p Let the triangular part R of the reduced QR factorization of the matrix L = [X, Y ] be [ ] R11 R R = 12, O R 22 where R 12 is full rank Then the positive singular values of the matrix T = R 22 (R 12 ) satisfy tan Θ(R(X), R(Y )) = [S + (T ), 0,, 0] Proof Let the reduced QR factorization of L be [ R11 R [ X Y ] = [ Q 1 Q 2 ] 12 O R 22 ], where the columns of [ Q 1 Q 2 ] are orthonormal, and R(Q 1 ) = X We directly obtain Y = Q 1 R 12 + Q 2 R 22 Multiplying by Q 1 on both sides of the above equality for Y, we get R 12 = Q 1 Y Similarly, multiplying by Q 2 we have R 22 = Q 2 Y Combining these equalities with Theorem 51 completes the proof 6 tan Θ in terms of the orthogonal projections onto subspaces In this section, we present T using orthogonal projectors only Before we proceed, we need a lemma and a corollary Lemma 61 [6] If U and V are unitary matrices, then for any matrix A we have (UAV ) = V A U Corollary 61 Let A C n n be a block matrix, such that [ ] D O12 A =, O 21 O 22 where D is a p q matrix and O 12, O 21, and O 22 are zero matrices Then [ ] D A O21 = O12 O22 12

15 Proof Let U(n) denote a family of unitary matrices of size n by n Let the SVD of D be U 1 ΣV1, where Σ is a p q diagonal matrix and U 1 U(p), V 1 U(q) Then there exist unitary matrices U 2 U(n p) and V 2 U(n q), such that [ ] [ ] [ ] U1 Σ O12 V A = 1 U 2 O 21 O 22 V2 Thus, [ ] [ ] [ ] A V1 Σ O12 U = 1 V 2 O 21 O 22 U2 [ ] [ ] [ ] V1 Σ O21 = U 1 V 2 O12 O22 U2 [ ] D Since D = V 1 Σ U1, we have A O21 = O12 O22 as desired Theorem 62 Let P X, P X and P Y be orthogonal projectors onto the subspaces X, X and Y, correspondingly Then the positive singular values S + (T ) of the matrix T = P X (P X P Y ) satisfy tan Θ(X, Y) = [,,, S + (T ), 0,, 0] Proof Let the matrices [ X X ] and [ Y Y ] be unitary, where R(X) = X and R(Y ) = Y By direct calculation, we obtain [ ] [ ] X X P X P Y [ Y Y ] = Y O O O X The matrix P X P Y can be written as [ X P X P Y = [ X X ] Y O O O ] [ Y Y ] (1) From Lemma 61 and Corollary 61, we have [ (X (P X P Y ) = [ Y Y ] Y ) O O O ] [ X X ] (2) In a similar way, P X P Y = [ X [ X X ] Y O O O 13 ] [ Y Y ] (3)

16 Let T = P X P Y (P X P Y ) and B = X Y ( X Y ), so [ ] [ ] [ ] X T = [ X X ] Y O (X Y ) O X O O O O X [ ] [ ] B O X = [ X X ] O O The singular values are invariant under unitary multiplications, therefore S + (T ) = S + (B) Finally, we show that P X P Y (P X P Y ) = P X (P X P Y ) The null space of P X P Y is the orthogonal sum Y ( Y X ) The range of (P X P Y ) is thus Y ( Y X ), which is the orthogonal complement of the null space of P X P Y Moreover, (P X P Y ) is an oblique projector, because (P X P Y ) = ( (PX P Y ) ) 2 which can be obtained by direct calculation using equality (2) The oblique projector (P X P Y ) projects onto Y along X Thus, we have P Y (P X P Y ) = (P X P Y ) Consequently, we can simplify T = P X P Y (P X P Y ) as T = P X (P X P Y ) X Figure 2: Geometrical meaning of T = P X (P X P Y ) To gain geometrical insight of T, in Figure 2 we choose a generic unit vector z We project z onto Y along X, then project onto the subspace X which is interpreted as T z The red segment in the graph is the image of T under all unit vectors It is straightforward to see that s(t ) = T = tan(θ) 14

17 Using Property 22 and the fact that the principal angles are symmetric with respect to the subspaces X and Y, the expressions P Y (P Y P X ) and P X (P X P Y ) can also be used in Theorem 62 Remark 63 According to Remark 44 and Theorem 62, we have ( P X (P X P Y ) = P X P Y (P X P Y ) = P X (P X P Y ) ) Furthermore, by the geometrical properties of the oblique projector (P X P Y ), it follows that (P X P Y ) = P Y (P X P Y ) = (P X P Y ) P X Therefore, we obtain Similarly, we have P X (P X P Y ) = P X P Y (P X P Y ) = (P Y P X )(P X P Y ) P Y (P Y P X ) = P Y P X (P Y P X ) = (P X P Y )(P Y P X ) = ( P Y (P Y P X ) ) From equalities above, we see that there are several different expressions for T in Theorem 62 Theorem 64 For subspaces X and Y, 1 If dim(x ) dim(y), we have P X P Y (P X P Y ) = P X (P X P Y ) = P X 2 If dim(x ) dim(y), we have (P X P Y ) P X P Y = (P X P Y ) P Y = P Y Proof Using the fact that X Y (X Y ) = I for dim(x ) dim(y) and combining identities (1) and (2), we obtain the first statement Identities (1) and (2), together with the fact that (X Y ) X Y = I for dim(x ) dim(y), imply the second statement Remark 65 From Theorem 64, for the case dim(x ) dim(y) we have P X (P X P Y ) = (P X P Y ) P X (P X P Y ) = (P X P Y ) P X Since the angles are symmetric, using the second statement in Theorem 64 we have (P X P Y ) P Y = (P X P Y ) P Y On the other hand, for the case dim(x ) dim(y) we obtain (P X P Y ) P Y = (P X P Y ) P Y and (P X P Y ) P X = (P X P Y ) P X To sum up, the following formulas for T in Table 2 can also be used in Theorem 62 An alternative proof for T = (P X P Y ) P Y is provided by Drmač in [14] for the particular case dim(x ) = dim(y) 15

18 Table 2: Different formulas for T : left for dim(x ) dim(y); right for dim(x ) dim(y) (P X P Y ) P X (P X P Y ) P Y (P X P Y ) P Y (P X P Y ) P X Remark 66 Finally, we note that our choice of the space = C n may appear natural to the reader familiar with the matrix theory, but in fact is somewhat misleading The principal angles (and the corresponding principal vectors) between the subspaces X and Y are exactly the same as those between the subspaces X X + Y and Y X + Y, ie, we can reduce the space to the space X + Y without changing PABS This reduction changes the definition of the subspaces X and Y and, thus, of the matrices X and Y that column-span the subspaces X and Y All our statements that use the subspaces X and Y or the matrices X and Y therefore have their new analogs, if the space X + Y substitutes for The formulas P X = I P X and P Y = I P Y, look the same, but the identity operators are different in the spaces X + Y and References [1] C Jordan, Essai sur la géométrie á n dimensions, Amer Math Monthly 3 (1875) [2] otelling, Relations between two sets of variates, Biometrika 28 (1936) [3] F Deutsch, The angle between subspaces of a ilbert space, in: Approximation theory, wavelets and applications (Maratea, 1994), volume 454 of NATO Adv Sci Inst Ser C Math Phys Sci, Kluwer Acad Publ, Dordrecht, 1995, pp [4] I C F Ipsen, C D Meyer, The angle between complementary subspaces, Amer Math Monthly 102 (1995) [5] A V Knyazev, M E Argentati, Majorization for changes in angles between subspaces, Ritz values, and graph Laplacian spectra, SIAM J Matrix Anal Appl 29 (2006/07) (electronic) [6] G W Stewart, J Sun, Matrix perturbation theory, Computer Science and Scientific Computing, Academic Press Inc, Boston, MA,

19 [7] J Sun, Perturbation of angles between linear subspaces, J Comput Math 5 (1987) [8] P Wedin, On angles between subspaces of a finite dimensional inner product space, in: B Kgstrm, A Ruhe (Eds), Matrix Pencils, volume 973 of Lecture Notes in Mathematics, Springer Berlin eidelberg, 1983, pp [9] C Davis, W M Kahan, The rotation of eigenvectors by a perturbation III, SIAM J Numer Anal 7 (1970) 1 46 [10] Ȧ Björck, G Golub, Numerical methods for computing angles between linear subspaces, Math Comp 27 (1973) [11] G Golub, Zha, The canonical correlations of matrix pairs and their numerical computation, in: Linear algebra for signal processing (Minneapolis, MN, 1992), volume 69 of IMA Vol Math Appl, Springer, New York, 1995, pp [12] A V Knyazev, A Jujunashvili, M Argentati, Angles between infinite dimensional subspaces with applications to the Rayleigh-Ritz and alternating projectors methods, J Funct Anal 259 (2010) [13] N Bosner, Z Drmač, Subspace gap residuals for Rayleigh-Ritz approximations, SIAM J Matrix Anal Appl 31 (2009) [14] Z Drmač, On relative residual bounds for the eigenvalues of a ermitian matrix, Linear Algebra Appl 244 (1996) [15] G W Stewart, Matrix algorithms Vol II, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2001 Eigensystems [16] A Edelman, B D Sutton, The beta-jacobi matrix model, the CS decomposition, and generalized singular value problems, Found Comput Math 8 (2008) [17] C C Paige, M A Saunders, Towards a generalized singular value decomposition, SIAM J Numer Anal 18 (1981) [18] C C Paige, M Wei, istory and generality of the CS decomposition, Linear Algebra Appl 208/209 (1994)

20 [19] P R almos, Two subspaces, Trans Amer Math Soc 144 (1969) [20] B D Sutton, Computing the complete CS decomposition, Numer Algorithms 50 (2009)