216 S. Chandrasearan and I.C.F. Isen Our results dier from those of Sun [14] in two asects: we assume that comuted eigenvalues or singular values are

Transcription

1 Numer. Math. 68: 215{223 (1994) Numerische Mathemati c Sringer-Verlag 1994 Electronic Edition Bacward errors for eigenvalue and singular value decomositions? S. Chandrasearan??, I.C.F. Isen??? Deartment of Comuter Science, Yale University, New Haven, CT 06520, USA Received July 19, 1993 Abstract. We resent bounds on the bacward errors for the symmetric eigenvalue decomosition and the singular value decomosition in the two-norm and in the Frobenius norm. Through dierent orthogonal decomositions of the comuted eigenvectors we can dene dierent symmetric bacward errors for the eigenvalue decomosition. When the comuted eigenvectors have a small residual and are close to orthonormal then all bacward errors tend to be small. Consequently it does not matter how exactly a bacward error is dened and how exactly residual and deviation from orthogonality are measured. Analogous results hold for the singular vectors. We indicate the eect of our error bounds on imlementations for eigenvector and singular vector comutation. In a more general context we rove that the distance of an aroriately scaled matrix to its orthogonal QR factor is not much larger than its distance to the closest orthogonal matrix. Mathematics Subject Classication (1991): 15A18, 15A23, 15A42, 65F15, 65F25, 65F35 1. Introduction We resent bounds on the bacward errors for the eigenvalue decomosition of real, symmetric matrices and for the singular value decomosition of real matrices. The bounds are given in the two-norm and in the Frobenius norm. We examine dierent bacward errors and give a osteriori bounds for each in terms of a residual and a deviation from orthogonality. We show that a small residual and deviation from orthogonality for one bacward error imlies the same for the other bacward errors. As a consequence it does not, in general, matter how exactly the bacward error is dened and how exactly the residual and the deviation from orthogonality are measured. We indicate how our results aect imlementations for the comutation of eigenvectors and singular vectors. In a more general context we rove that the distance of an aroriately scaled matrix to its orthogonal QR factor is not much larger than its distance to the closest orthogonal matrix.? The wor resented in this aer was suorted by NSF grant CCR ?? shiv@cs.yale.edu??? isen@cs.yale.edu age 215 of Numer. Math. 68: 215{223 (1994)

2 216 S. Chandrasearan and I.C.F. Isen Our results dier from those of Sun [14] in two asects: we assume that comuted eigenvalues or singular values are given; and we do not assume that the comuted eigenvectors or singular vectors form an orthonormal basis. In fact, our main interest is recisely the dierence in bacward errors resulting from dierent ways of measuring the deviation from orthogonality. Our results are required for an analysis of inverse iteration [1]: On the one hand we want the freedom to use whatever bacward error aears most convenient at dierent oints in the analysis but on the other hand we need to be assured that the result of the analysis is not aected by the articular choice of error Bacward errors We start by determining bacward errors for eigenvalue decomositions A = V V T, where is diagonal and V is orthogonal. In order to assess the accuracy of the comuted eigenvalue matrix ^ and eigenvector matrix ^V in the bacward sense, one would ideally lie to to show that ^ and ^V reresent an exact eigenvalue decomosition of some symmetric matrix. We dene the two obvious bacward errors ^E and E by A + ^E = T ^V ^ ^V and A + E = ^V ^ ^V?1, resectively (Sect. 2.1). If ^E or E is small then ^ and ^V reresent a decomosition of a nearby matrix. However, while it is usually easy to ensure that ^ is diagonal, it is generally imossible to numerically comute a matrix ^V that is orthogonal. Therefore A + E = ^V ^ ^V?1 is not symmetric, while A + ^E = ^V ^ ^V T is not an eigenvalue decomosition. Hence the two obvious bacward errors do not reresent an eigenvalue decomosition of some symmetric matrix. Alternatively, one can extract an orthogonal basis W from ^V through a decomosition ^V = W Z, and dene a symmetric bacward error EW by A + E W = W ^W T. Then ^ and W reresent an eigenvalue decomosition of some symmetric matrix A + E W (Sect. 2.2). If ^V is close to W and if EW is small then ^V and ^ are close to a decomosition that reresents an eigenvalue decomosition of a nearby symmetric matrix A + E W. This corresonds exactly to the denition of stability in [12] for an algorithm that comutes and V Notation The norm 2 denotes the Euclidean norm, F denotes the Frobenius norm, and reresents either 2 or F. The n n identity matrix is denoted by I or I n, and its ith column by e i. We mae use of the fact [8, Corollary ] that for matrices A and B AB A 2 B: This reresents a tighter bound for the Frobenius norm than AB F A F B F. As a consequence, our error bounds for the two-norm and the Frobenius norm turn out to be identical. age 216 of Numer. Math. 68: 215{223 (1994)

3 Bacward errors The symmetric eigenvalue decomosition Let A be a real symmetric matrix of order n with eigenvalue decomosition A = V V T, where the eigenvector matrix V is real orthogonal, and the eigenvalue matrix is real and diagonal. Denote the matrix of comuted real eigenvalues by ^ and the matrix of comuted real eigenvectors by ^V The obvious bacward errors Let r A ^V? ^V ^; o I? ^V T ^V = I? ^V ^V T denote the residual and deviation from orthogonality, resectively, and dene a bacward error ^E by A + ^E = ^V ^ ^V T : Then ^E is small if both residual and deviation from orthogonality are small, ^E r ^V 2 + o A 2 : The bound is tight when ^V is orthogonal. If ^V is not orthogonal then ^V ^ ^V T does not corresond to an eigenvalue decomosition of A + ^E. For non-singular ^V one can also dene a bacward error E as in [9], A + E = ^V ^ ^V?1 : If both residual and deviation from orthogonality are small, then the comuted decomosition reresents an eigenvalue decomosition of a nearby matrix, for E r 1? o ; assuming o < 1, as was already roved in a slightly dierent form in [9]. The bound is tight when ^V is orthogonal. If ^V is not orthogonal then A + E may not be symmetric Bacward error based on orthogonal decomositions Factor ^V = W Z, where W is orthogonal, and dene a symmetric bacward error E W by A + E W = W ^W T : Denote by W r AW? W ^; W o ^V? W the residual and deviation from orthogonality, resectively. Then the decomosition ^V ^ ^V T is close to an eigenvalue decomosition of some symmetric matrix if the deviation from orthogonality is small, ^E? E W W o ^2 (1 + ^V 2 ): age 217 of Numer. Math. 68: 215{223 (1994)

4 218 S. Chandrasearan and I.C.F. Isen If, in addition, the residual W r = E W is also small then ^V and ^ are close to an eigenvalue decomosition of a nearby symmetric matrix. Although the above bound holds for any orthogonal decomosition, W o is not necessarily small for any W. For instance, if ^V is almost orthogonal then the distance U1 o of ^V from its left singular vector matrix U1 is large unless the right singular matrix is close to the identity matrix. Fortunately there are two decomositions that give rise to small W o for almost orthogonal matrices. The rst one is the olar factorisation ^V = P H, where the olar factor P is the closest orthogonal matrix to ^V [2, 5, 7]. Hence P o W o for any orthogonal matrix W. The olar decomosition is related to the singular value decomosition ^V = U1 U2 T by P = U 1 U2 T, where U 1 and U 2 are orthogonal and is diagonal [7, Examle 7.4.6]. The bacward error in [4], for instance, is based on the olar decomosition. The second decomosition is the QR decomosition ^V = QR [3]. When ^V is aroriately scaled then the distance Q o of ^V to its orthogonal QR factor Q is small (Sect. 2.5) Relation between residual and accuracy of comuted eigenvalues If =? ^ is the accuracy of the comuted eigenvalues ^ then the residual W r as large, W r : is at least This follows because the bacward error E W is symmetric, and one can aly Weyl's Theorem for the two-norm [11, Fact 1-11,Theorem ] and the Wielandt-Homan Inequality for the Frobenius norm [11, Fact 1-11]. The same is true for the residual r, rovided the deviation from orthogonality is taen into account, r (1? o ): This follows from ^V? ^V ^? ^= ^V?1 ; assuming that ^V is non-singular. The two-norm version of this result is due to Kahan [11, Theorem ], [10], [13, Lemma IV.5.3], while the Frobenius norm version is due to Sun and Zhang [13, Theorem IV.5.5]. Therefore, the accuracy of an almost orthogonal set of comuted eigenvectors is limited by the accuracy of the comuted eigenvalues. Conversely, if both residual and deviation from orthogonality are small then the comuted eigenvalues must be close to the exact eigenvalues. Imlementation issues. When eigenvectors are comuted from highly accurate eigenvalues, such as in inverse iteration or erfect-shift QR, it maes sense to use r as a stoing criterion for the eigenvector comutation. age 218 of Numer. Math. 68: 215{223 (1994)

5 Bacward errors Partial eigenvalue decomositions Suose a subset of n eigenvalues and eigenvectors is to be comuted. For the matrix ^V with columns and the diagonal matrix ^ (where ^Vn = ^V and ^n = ^), the residual and deviation from orthogonality, resectively, are r A ^V? ^V ^ ; o I? ^V T ^V : Wilinson [15, Sect. 3.53] denes a bacward error for the case = 1, which Stewart and Sun [13, Theorem IV.1.13] extend to > 1 as follows. If U is any real n matrix with U T ^V = I and then E = ( ^V ^? A ^V )U T (A + E ) ^V = ^V ^ : If U is the seudo-inverse of ^V and if o < 1 then U 2 1= 1? o. Hence the bacward error E is bounded above by E r 1? o : In the case of bacward errors based on orthogonal decomositions ^V = W Z, where the n matrix W has orthonormal columns and the matrix Z is, one can set U = W, so E W = (W ^? AW )W T : As in the case of the full eigenvalue decomosition, the bacward error E W bounded by the residual, E W = W, where r is The deviation from orthonormality is W r AW? W ^ : W o W? ^V : Bacward errors based on an orthogonal decomositions can be used, for instance, to determine the accuracy of inverse iteration. There the comuted vectors associated with close eigenvalues are exlicitly orthogonalised (in nite recision) via a QR decomosition. Because the vectors ^Q are numerically orthonormal, the bacward error deends (to rst order) only on the residual A ^Q? ^Q ^. Imlementation issues. An exlicit orthogonalisation of the comuted vectors, by the modied Gram-Schmidt algorithm as in inverse iteration for instance, does not increase the residual signicantly if the vectors are already retty orthonormal to start with, W r r + o (A 2 + ^ 2 ): age 219 of Numer. Math. 68: 215{223 (1994)

6 220 S. Chandrasearan and I.C.F. Isen 2.5. Relations between dierent orthogonality measures Section 4 shows that for aroriately scaled vectors ^V the dierent measures of orthogonality are essentially equivalent; and that the orthonormal QR factor is about as close to ^V as the closest matrix with orthonormal columns, the olar factor. As a consequence, P o Q o 5 P o ; whenever ^V e i = 1, 1 i. Moreover the measures o and W o equivalent: When W is the olar factor then [6] are almost These bounds imly for the QR factor assuming that ^V e i = 1. P o o P o (1 + ^V 2 ): 1 5 Q o o Q o (1 + ^V 2 ); Imlementation issues.the orthonormal QR factor Q is essentially as close to ^V as the closest orthonormal matrix. If R is the triangular QR factor, then the distance of ^V to the closest orthogonal matrix is bounded by R? I. 3. The singular value decomosition Let B be a real square matrix of order m with singular value decomosition (SVD) B = XY T, where the singular vector matrices X and Y are real orthogonal and the singular value matrix is real diagonal The obvious bacward error Denote by r maxfb T ^X? ^Y ^; B ^Y? ^X ^g the residual and by T o maxfi? ^X ^X; I? T ^Y ^Y g the deviation from orthogonality, and dene a bacward error ^F by B + ^F = ^X ^ ^Y T : Then ^F r minf ^X2 ; ^Y 2 g + o B 2 : age 220 of Numer. Math. 68: 215{223 (1994)

7 Bacward errors Bacward errors based on orthogonal decomositions Factor ^X = Wx Z x and ^Y = Wy Z y with W x and W y orthogonal, and dene a bacward error F W by B + F W T = W x ^W y : Denote by W r the residual and by maxfbw y? W x ^; B T W x? W y ^g W o maxf ^X? Wx ; ^Y? Wy g the deviation from orthogonality. The comuted quantities ^X, ^Y and ^ are close to the SVD of some matrix if the deviation from orthogonality is small, ^F? F W W o ^2 (1 + minf ^X2 ; ^Y 2 g): If, in addition, the residual is also small then ^X, ^Y and ^ are close to the SVD of a nearby matrix Rectangular matrices If B is a m l matrix with m l its SVD can be written as B = X Y 0 T ; where X is orthogonal of order m, Y is orthogonal of order l, and is diagonal of order l. With? r = maxfb T ^X? ^Y ^ 0 ; B ^Y? ^X ^ g 0 and? W r = maxfb T W x? W y ^ 0 ; BWy? W ^ x g 0 all results for square matrices continue to hold. 4. Relation between QR and olar factors Let ^V be a n matrix of ran with olar decomosition ^V = P H and QR decomosition ^V = Q R, where R has ositive diagonal elements. If ^V e i = 1, 1 i, then ^V? Q 5 ^V? P 2 : age 221 of Numer. Math. 68: 215{223 (1994)

8 222 S. Chandrasearan and I.C.F. Isen Proof. We distinguish the two cases ^V? P 2 < 2=5 and ^V? P 2 2=5. If ^V? P 2=5 then ^V? Q 2 5 ^V? P 2 : Now assume that ^V? P 2 < 2=5. It suces to wor with the triangular QR factor of ^V as ^V? Q = Q (R? I ) = R? I : Let the (l+1)st column be the one that attains the largest norm max 1i (R? I )e i, and consider only the leading rincial submatrix of order l + 1 of R : Rl r R l+1 = ; r where r = 1, and R l is of order l. If we set r l+1 then ^V? Q = R? I rl+1? e +1 = 2(1? ): In order to nd an uer bound on r l+1? e +1 we need to nd a lower bound on. To this end, we rst exress the smallest singular value min ( R) of Il r R in terms of and then bound min ( R) below in terms of 2 ^V? P 2. As a consequence we get a lower bound on in terms of 2, and hence the desired uer bound on r l+1? e l+1. The singular values of R equal one and q1 1? 2. Thus, min ( R) = q1? 1? 2 : Let the singular value decomosition of R l be R l = U 1 U2 T, where U 1 and U 2 are orthogonal. Then U1 U R l+1 = U T T r 2 l+1 ; 1 U1 I U R = U Tr T 1 l+1 : 1 According to the erturbation theory for singular values [3, Corollary 8.3.2], i (R l+1 )? i ( R) = I i U 0 1 T r l+1? i U 0 1 T r l+1?i 2 : The interlacing of the singular values of R l among those of R [3, Corollary 8.3.3] imlies age 222 of Numer. Math. 68: 215{223 (1994)

9 Bacward errors 223?I 2 = max j Hence j j (R l )? 1j max j j j (R )? 1j = max j j ( ^V )? 1j = ^V?P 2 = 2 : j i (R l+1 )? 2 i ( R) i (R l+1 ) + 2 : Furthermore, the singular values of R satisfy 1? 2 i (R ) ; and due to interlacing the singular values of R l+1 also satisfy Therefore 1? 2 2 min ( R). 1? 2 i (R l+1 ) : Combining the last inequality with min ( R) = q1? 1? 2 results in the lower bound 1?8 2 2 under the condition that 2 < 2=5. Hence r l+1?e l+1 2(1? ) 42. Acnowledgement. We would lie to than Stan Eisenstat for helful discussions and for maing suggestions that imroved the manuscrit. References 1. Chandrasearan, S., Isen, I. (1993): Finite recision analysis of inverse iteration. Research Reort 919, Deartment of Comuter Science, Yale University, Yale 2. Fan, K., Homan, J. (1955): Some metric inequalities in the sace of matrices. Proc. Amer. Math. Soc. 6, 111{ Golub, G., Van Loan, C. (1989): Matrix comutations. The Johns Hoins Press, Baltimore, Maryland, USA 4. Gu, M. (1993): Three roblems in numerical linear algebra. PhD thesis, Deartment of Comuter Science, Yale University 5. Heindl, G. (1990): An ecient method for imroving orthonormality of nearly orthonormal sets of vectors. Contributions to comuter arithmetic and self-validating numerical methods,. 83{91 6. Higham, N. (1993): The matrix sign decomosition and its relation to the olar decomosition. Manuscrit, Det. of Mathematics, University of Manchester, Manchester, UK 7. Horn, R., Johnson, C. (1985): Matrix analysis. Cambridge University Press, Cambridge 8. Horn, R., Johnson, C. (1991): Toics in matrix analysis. Cambridge University Press, Cambridge 9. Jessu, E., Isen, I. (1992): Imroving the accuracy of inverse iteration. SIAM J. Sci. Stat. Comut. 13, 550{ Kahan, W. (1967): Inclusion theorems for clusters of eigenvalues of Hermitian matrices. Tech. Reort CS42, Comuter Science Deartment, University of Toronto, Toronto 11. Parlett, B. (1980): The symmetric eigenvalue roblem. Prentice Hall, Englewood Clis, New Jersey, USA 12. Stewart, G. (1973): Introduction to matrix comutations. Academic Press, New Yor, New Yor, USA 13. Stewart, G., Sun, J. (1990): Matrix erturbation theory. Academic Press, San Diego, California, USA 14. Sun, J. (1992): Bacward erturbation analysis of certain characteristic subsaces. Tech. Reort LiTH-MAT-R , Deartment of Mathematics, Linoing University, Linoing, Sweden 15. Wilinson, J. (1965): The algebraic eigenvalue roblem. Oxford University Press, Oxford age 223 of Numer. Math. 68: 215{223 (1994)