Eigenvalue and Eigenvector Problems An attempt to introduce eigenproblems Radu Trîmbiţaş Babeş-Bolyai University April 8, 2009 Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 1 / 23
Eigenvalues and Eigenvectors A C m m ; a nonzero vector x C m is an eigenvector of A, and λ C is its corresponding eigenvalue, if The set is the spectrum of A Applications: Ax = λx. Λ(A) = {λ C λ eigenvalue of A} Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 2 / 23
Eigenvalues and Eigenvectors A C m m ; a nonzero vector x C m is an eigenvector of A, and λ C is its corresponding eigenvalue, if The set is the spectrum of A Applications: Ax = λx. Λ(A) = {λ C λ eigenvalue of A} algorithmically eigenvalue analysis simplifies problems by reducing them to a collection of scalar problems Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 2 / 23
Eigenvalues and Eigenvectors A C m m ; a nonzero vector x C m is an eigenvector of A, and λ C is its corresponding eigenvalue, if The set is the spectrum of A Applications: Ax = λx. Λ(A) = {λ C λ eigenvalue of A} algorithmically eigenvalue analysis simplifies problems by reducing them to a collection of scalar problems physically study of resonance and stability Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 2 / 23
The Eigenvalue Decomposition Eigenvalue decomposition of A: A = X ΛX 1 where X is nonsingular and Λ is diagonal or equivalently AX = ΛX with eigenvectors as columns of X and eigenvalues on diagonal of Λ. Such a factorization does not always exist In eigenvector coordinates, A is diagonal: Ax = b = X 1 b = Λ(X 1 x) Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 3 / 23
Multiplicity The eigenvectors corresponding to a single eigenvalue (plus the zero vector) form an eigenspace E λ Dimension of E λ = dim(null(a λi )) = geometric multiplicity of λ The characteristic polynomial of A is p A (z) = det(zi A) = (z λ 1 )(z λ 2 )... (z λ m ) λ is eigenvalue of A p A (λ) = 0 λ is an eigenvalue x = 0, λx Ax = 0 λi A is singular det(λi A) = 0 Multiplicity of a root to p A = algebraic multiplicity of λ Any matrix A has m eigenvalues, counted with algebraic multiplicity Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 4 / 23
Similarity Transformations The map A X 1 AX is a similarity transformation of A A and B are similar if there is a similarity transformation B = X 1 AX A and X 1 AX have the same characteristic polynomials, eigenvalues, and multiplicities: The characteristic polynomials are the same: p X 1 AX (z) = det(zi X 1 AX ) = det(x 1 (zi A)X ) = det(x 1 ) det(zi A) det(x ) = det(zi A) = p A (z) Therefore, the algebraic multiplicities are the same If E λ is eigenspace for A, then X 1 E λ is eigenspace for X 1 AX, so geometric multiplicities are the same Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 5 / 23
Theorem Algebraic Multiplicity Geometric Multiplicity Proof. Let n first columns of V be an orthonormal basis of the eigenspace for λ(n is the geometric multiplicity of A). Extend V to square unitary V, and form [ ] B = V λi C AV = 0 D Since det(zi B) = det(zi λi ) det(zi D) = (z λ) n det(zi D) the algebraic multiplicity of λ (as eigenvalue of B) is n. A and B are similar; so the same is true for λ of A Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 6 / 23
Defective and Diagonalizable Matrices If the algebraic multiplicity for an eigenvalue > its geometric multiplicity, it is a defective eigenvalue If a matrix has any defective eigenvalues, it is a defective matrix A nondefective matrix has equal algebraic and geometric multiplicities for all eigenvalues The matrix A is nondefective A = X ΛX 1 ( =) If A = X ΛX 1, A is similar to Λ and has the same eigenvalues and multiplicities. But Λ is diagonal and thus nondefective. (= ) Nondefective A has m linearly independent eigenvectors. Take these as the columns of X, then A = X ΛX 1 Thus, nondefective diagonalizable Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 7 / 23
Determinant and Trace The trace of A is tr(a) = m j=1 a jj The determinant and the trace are given by the eigenvalues: det(a) = m j=1 λ j, tr(a) = m λ j, j=1 since det(a) = ( 1) m det( A) = ( 1) m p A (0) = m j=1 λ j and p A (z) = det(zi A) = z m m j=1 p A (z) = (z λ 1 )... (z λ m ) = z m a jj z m 1 + m j=1 λ j z m 1 + Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 8 / 23
Canonical Forms (Jordan canonical form) A S nonsingular, such that A = SJS 1, J is block diagonal, with J = diag(j m1 (λ 1 ),..., J mk (λ k )) and J mi (λ i ) = λ i 1 0......... 1 0 λ i m i m i J is unique, up to permutations of its diagonal blocks, m i = m J p (λ) is a Jordan block with eigenvalue λ with algebraic multiplicity p if some m i = 1, and λ i is an eigenvalue of only that one Jordan block, then λ i is simple if all m i = 1, A is diagonalizable, otherwise defective Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 9 / 23
Canonical Forms-continued A Schur factorization of a matrix A is a factorization A = QTQ with unitary Q and upper-triangular T. The eigenvalues of A are the diagonal entries of T. A real Schur factorization of a real matrix A is a factorization A = QTQ T, where Q is orthogonal, and T is quasi-upper triangular (block upper triangular with 1 1 and 2 2 blocks on the diagonal). T eigenvalues are eigenvalues of the diagonal blocks: the 1 1 blocks correspond to real eigenvalues, and 2 2 blocks correspond to complex conjugate pairs of eigenvalues. Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 10 / 23
Unitary Diagonalization and Schur Factorization A matrix A is unitary diagonalizable if, for some unitary matrix Q, A = QΛQ A hermitian matrix is unitary diagonalizable, with real eigenvalues (because of the Schur factorization, see below) A is unitarily diagonalizable A is normal (AA = A A ) (a classical result) Summary, Eigenvalue-Revealing Factorizations Diagonalization A = X ΛX 1 (nondefective A) Unitary diagonalization A = QΛQ (normal A) Unitary triangularization (Schur factorization) A = QTQ (any A) Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 11 / 23
Theorem Every square matrix A C m m has a Schur factorization. Proof. Induction on m. m = 1, trivial; m 2. Let x be a normalized ev of A with corresponding ew λ; x the first column of a unitary matrix U. It is easy to check that U AU = [ λ B 0 C By the inductive hypothesis, a Schur factorization of C, VTV = C. [ ] 1 0 Q = U is unitary, and 0 V [ ] Q λ BV AQ = 0 T This is the Schur decomposition we seek. ]. Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 12 / 23
Eigenvalue Algorithms The most obvious method is ill-conditioned: Find roots of p A (λ) Instead, compute Schur factorization A = QTQ by introducing zeros However, this can not be done in a finite number of steps: Any eigenvalue solver must be iterative To see this, consider a general polynomial of degree m p(z) = z m + a m 1 z m 1 + + a 1 z + a 0 There is no closed-form expression for the roots of p: (Abel, 1842) In general, the roots of polynomial equations higher than fourth degree cannot be written in terms of a finite number of operations Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 13 / 23
Eigenvalue Algorithms (continued) However, the roots of p are the eigenvalues of the companion matrix 0 a 0 1 0 a 1 1 0 a 2 A = 1....... 0 am 2 1 a m 1 Therefore, in general we cannot find the eigenvalues of a matrix in a finite number of steps (even in exact arithmetic) In practice, algorithms available converge in just a few iterations Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 14 / 23
Perturbation Theory (Bauer-Fike Theorem) Suppose A C m m is diagonalizable with A = V ΛV 1, and let δa C m m arbitrary. Then every eigenvalue of A + δa, λ i lies in at least one of the m circular disks in the complex plane of radius κ(v ) δa 2 centered at the eigenvalues of A, where κ is the 2-norm condition number. λ j λ j κ(v ) δa 2 If A is normal, for each eigenvalue λ j of A + δa, there is an eigenvalue λ j of A such that λ j λ j δa 2 Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 15 / 23
Schur Factorization and Diagonalization Compute Schur factorization A = QTQ by transforming A with similarity transformations which converge to T as j. Qj... Q2 Q1 A Q 1 Q 2... Q j }{{}}{{} Q Q Note: Real matrices might need complex Schur forms and eigenvalues (or a real Schur factorization with 2 2 blocks on diagonal) For hermitian A, the sequence converges to a diagonal matrix Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 16 / 23
Two Phases of Eigenvalues Computations General A: First to upper-hessenberg form, then to upper-triangular Phase 1 Phase 2 A = A H T Hermitian A: First to tridiagonal form, then to diagonal Phase 1 Phase 2 A = A T D Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 17 / 23
Hessenberg/Tridiagonal Reduction - phase 1 - We apply orthogonal transforms to convert A into an upper Hessenberg matrix H It requires O(m 3 ) flops Can be much longer than the second phase, although the latter requires (theoretically) an infinite number of steps Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 18 / 23
Introducing Zeros by Similarity Transformations Try computing the Schur factorization A = QTQ by applying Householder reflectors from left and right that introduce zeros (a bad idea): A Q 1 0 0 0 0 Q1 A Q 1 Q1 AQ 1 The right multiplication destroys the zeros previously introduced We already knew this would not work, because of Abel s theorem However, the subdiagonal entries typically decrease in magnitude Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 19 / 23
The Hessenberg Form Instead, try computing an upper Hessenberg matrix H similar to A: Q 1 0 Q 1 0 0 A Q1 A Q1 AQ 1 This time the zeros we introduce are not destroyed Continue in a similar way with column 2: Q 2 Q 2 0 0 Q1 AQ 1 Q2 Q 1 AQ 1 Q2 Q 1 AQ 1Q 2 Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 20 / 23
The Hessenberg Form After m 2 steps, we obtain the Hessenberg form: Qm 2... Q2 Q1 A Q 1 Q 2... Q m 2 = H = }{{}}{{} Q Q For hermitian A, zeros are also introduced above diagonals 0 0 0 Q 1 0 Q 1 0 0 A Q1 A Q1 AQ 1 producing a tridiagonal matrix T after m 2 steps Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 21 / 23
Householder Reduction to Hessenberg Householder Reduction to Hessenberg form for k := 1 to m 2 do x := A k+1:m,k ; v k := sign(x 1 ) x 2 e 1 + x; v k := v k / v k 2 ; A k+1:m,k:m := A k+1:m,k:m 2v k (v k A k+1:m,k:m); A 1:m,k+1:m := A 1:m,k+1:m 2 (A 1:m,k+1:m v k ) v k Operation count (not twice Householder QR): m k=1 [4(m k) 2 + 4m(m k)] 10 3 m3 For hermitian A, operation count is twice QR divided by two 4m 3 /3 Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 22 / 23
Stability of Householder Hessenberg The Householder Hessenberg reduction algorithm is backward stable: Q H Q = A + δa, δa A = O(ɛ machine) where Q is an exactly unitary matrix based on ṽ k Radu Trîmbiţaş ( Babeş-Bolyai University) Eigenvalue and Eigenvector Problems April 8, 2009 23 / 23