Lecture 2: Eigenvalues, eigenvectors and similarity The single most important concept in matrix theory. German word eigen means proper or characteristic. KTH Signal Processing 1 Magnus Jansson/Emil Björnson Example: Stability of linear systems Consider a linear discrete time homogenous system: x(n +1)=Ax(n) If the spectral radius of A is greater than 1, and x(n) is not orthogonal to the corresponding eigenvectors, x(n +1)will grow in the direction of the unstable modes. KTH Signal Processing 3 Magnus Jansson/Emil Björnson Definition Consider: Square matrix A Mn If there exists x C n ( x 0) and λ C such that Ax = λx λ is an eigenvalue of A x is an eigenvector of A associated with λ More terminology: Spectrum of A: Set of all eigenvalues. Notation: σ(a). Eigenspace of A: Set of all eigenvectors. Spectral radius of A: ρ(a) =max{ λ : λ σ(a)}. KTH Signal Processing 2 Magnus Jansson/Emil Björnson Example: Filter output power maximization Let y(t) be a vector of observations at time t, and let z(t) =x T y(t) be the output of a filter with filter coefficients in the vector x. The mean output power is E{z 2 (t)} =E{ ( x T y(t) )( y T (t)x ) } = x T E{y(t)y T (t)}x = x T Ryx Assume we want to find the filter that maximizes the output power under a normalizing constraint on the filter gain max x x T Ryx subject to x T x =1 The solution to this is given by the unit length eigenvector corresponding to the largest eigenvalue of Ry. KTH Signal Processing 4 Magnus Jansson/Emil Björnson
What about zero? Recall: If exists x C n ( x 0) and λ C such that Ax = λx λ is an eigenvalue of A x is an eigenvector of A associated with λ Question: Why x 0? Answer: We always have A0 = λ0 (uninteresting solution!) However: λ =0is an important case: Ax =0=0x In fact: A Mn is singular iff it exists x 0such that Ax =0x =0or, equivalently, iff 0 σ(a) KTH Signal Processing 5 Magnus Jansson/Emil Björnson How to find an eigenvalue? Rewrite eigenvalue definition (A Mn): λx Ax =0 (λi A)x =0 Observation: Eigenvalues make (λi A) singular, det(λi A) =0. Definition: Characteristic polynomial is pa(t) =det(ti A). pa(t) is polynomial of degree n: Has n solutions/roots to pa(t) =0. Conclusions: These roots are the eigenvalues of A. A Mn has n (complex) eigenvalues. Some eigenvalues may have (algebraic) multiplicity! KTH Signal Processing 7 Magnus Jansson/Emil Björnson Polynomials of matrices Consider: Scalar polynomial p(t) =akt k + ak 1t k 1 +...+ a0. Define: Matrix polynomial p(a) =aka k + ak 1A k 1 +...+ a0i for A Mn. Theorem: If λ is an eigenvalue of A and x the associated eigenvector, p(a)x = p(λ)x. Thus, x is also eigenvector of p(a) associated with eigenvalue p(λ). KTH Signal Processing 6 Magnus Jansson/Emil Björnson Trace and determinant Definition: Trace is tr(a) = n i=1 a ii Definition: Determinant is det(a) =[Laplace expansion in 0.3.1] Theorem: Expressed using eigenvalues as tr(a) = n i=1 λi, det(a) = n i=1 λi. Observation: Coefficients in characteristic polynomial pa(t) =t n + an 1t n 1 +...+ a0 where an 1 = tr(a), a0 =( 1) n det(a). Formulas exist for all ak; see the book. KTH Signal Processing 8 Magnus Jansson/Emil Björnson
Similarity Consider: A Mn, B Mn Definition: B is similar to A if there exists a nonsingular S Mn such that A = S 1 BS Notation: B A Transformation A S 1 AS called similarity transformation by the similarity matrix S. KTH Signal Processing 9 Magnus Jansson/Emil Björnson Diagonalizable matrices Definition: A Mn is diagonalizable if it is similar to a diagonal matrix. Theorem: A is diagonalizable iff A has n linearly independent eigenvectors. KTH Signal Processing 11 Magnus Jansson/Emil Björnson Equivalence class Similarity is reflexive A A symmetric B A implies A B transitive C B and B A imply C A Divides all matrices into (disjoint) equivalence classes: Each class has a representative matrix A. The class includes all matrices similar to A. Theorem: If B A, pb(t) =pa(t). B and A have the same eigenvalues (counting multiplicity). KTH Signal Processing 10 Magnus Jansson/Emil Björnson Linearly independent eigenvectors Assume: λ1,λ2,...,λn are distinct eigenvalues of A Mn xi is eigenvector associated with λi Theorem: {x1,x2,...,xn} is a linearly independent set. Conclusion: If A Mn has n distinct eigenvalues A is diagonalizable. (The converse is not true.) KTH Signal Processing 12 Magnus Jansson/Emil Björnson
Simultaneous diagonalization Definition: A, B Mn commute if AB = BA Definition: Two diagonalizable matrices A, B Mn are simultaneously diagonalizable if there exists a single similarity matrix S Mn diagonalizing both A and B. Theorem: A, B commute iff they are simultaneously diagonalizable. KTH Signal Processing 13 Magnus Jansson/Emil Björnson How to find the eigenvector? Rewrite eigenvalue definition (A Mn): λx Ax =0 (λi A)x =0 Observation: x lies in the nullspace of λi A. Calculate eigenvectors: Solve (λi A)x =0for eigenvalue λ. System of equations: Use Gauss elimination. Eigenvector is non-unique: Any scaling (x 0) Nullspace can have large dimension. KTH Signal Processing 15 Magnus Jansson/Emil Björnson Eigenvalues of products Assume: A Mm,n and B Mn,m with m n. Theorem: pba(t) =t n m pab(t) BA has the same eigenvalues as AB plus n m additional eigenvalues at zero. If m = n and A (or B) is nonsingular, AB is similar to BA. KTH Signal Processing 14 Magnus Jansson/Emil Björnson Eigenspace The set of all eigenvectors satisfying Ax = λx for a given λ σ(a) is called the eigenspace of A corresponding to λ. The eigenspace, together with the zero vector, is a subspace of C n and it is exactly the nullspace of λi A. KTH Signal Processing 16 Magnus Jansson/Emil Björnson
Multiplicity Algebraic multiplicity: Multiplicity of the corresponding root of the characteristic polynomial. Geometric multiplicity: Number of linearly independent eigenvectors associated with the eigenvalue. Theorem: Algebraic multiplicity Geometric multiplicity Definition: If strict inequality for some eigenvalue, the matrix is defective. Theorem: A is diagonalizable iff it is not defective. KTH Signal Processing 17 Magnus Jansson/Emil Björnson Transpose and conjugate transpose Transpose: A and A T have same eigenvalues. Left/right eigenvectors are interchanged and complex conjugated. Conjugate transpose: Eigenvalues of A are complex conjugates of eigenvalues of A. Left/right eigenvectors are interchanged. KTH Signal Processing 19 Magnus Jansson/Emil Björnson Left eigenvectors A nonzero vector y C n is a left eigenvector of A Mn if y A = μy. Observe that μ σ(a). Theorem (Biorthogonality): Let y A = μy and Ax = λx. Then, if μ λ we have y x =0. Observation: If A is Hermitian (A = A ), x = y for same eigenvalue. Biorthogonality implies that A has n pair-wise orthogonal eigenvectors of (at least if eigenvalues are distinct, more later). KTH Signal Processing 18 Magnus Jansson/Emil Björnson