Data Analysis and Manifold Learning Lecture 2: Properties of Symmetric Matrices and Examples

Transcription

1 Data Analysis and Manifold Learning Lecture 2: Properties of Symmetric Matrices and Examples Radu Horaud INRIA Grenoble Rhone-Alpes, France

2 Outline of Lecture 2 Basic definitions, eigen decomposition, LU and Cholesky matrix factorizations; Spectral decomposition, powers, inverse, exponential; Geometric interpretation; The Raleigh-Ritz theorem and extensions; Computing eigenvalues and eigenvectors in practice: power method, inverse power method, and shifted inverse power method;

3 Material for This Lecture R. A. Horn and C. R. Johnson. Matrix Analysis. Chapter 4: Hermitian and symmetric matrices. G. H. Golub and C. F. Van Loan. Matrix Computations. Chapter 8: The symmetric eigenvalue problem. Chapter 9: Lanczos methods. Software: written in Fortran77!

4 Some Basic Definitions Symmetry of a D D matrix: A = A Eigen decomposition: A = UΛU with the properties: UU = U U = I D det(u) = ±1 All the eigenvalues are real numbers: λ min = λ 1... λ i... λ D = λ max A is referred to as a real symmetric matrix; If λ 1 0 then it is a positive semi-definite symmetric matrix If λ 1 > 0 then it is a positive definite symmetric matrix Symmetric matrices are nondefective: the algebraic multiplicity of each eigenvalue is equal to its geometric multiplicity.

5 Spectral Decomposition, Deflation, Powers, Exponential A symmetric matrix can be written as A = D i=1 λ iu i u i where {u i } i=d i=1 are the eigenvectors or, equivalently, the column vectors of U. The transformation à = A λ ku k u k Note that Ãu k = 0. A 2 = UΛU UΛU = UΛ 2 U More generally: A k = UΛ k U is known as a deflation. The matrices A, A 2,..., A k have the same eigenvectors {u i } and eigenvalues λ, λ 2,..., λ k. Matrix exponential: e A = k=0 Ak k! We have: e A = UDiag[e λ 1... e λ i... e λ D]U Hence, matrix e A has the same eigenvectors as A and eigenvalues e λ.

6 The Inverse of a Symmetric Matrix The case of a non-singular symmetric matrix: A 1 = UΛ 1 U. Spectral decomposition: A 1 = D i=1 1 λ i u i u i The matrices A 1, A 2,..., A k have eigenvectors u i and eigenvalues λ 1 i, λ 2 i,..., λ k i

7 The Pseudoinverse of a Singular Symmetric Matrix If a symmetric matrix is singular, it has as eigenvalue λ = 0 with multiplicity m > 0. We rearrange the eigenvalue-eigenvector pairs and we retain the non-zero pairs, such that: A = Ũ ΛŨ with: Λ = Diag[λ 1... λ D m ] and Ũ = [u 1... u D m ]. Ũ is a D (D m) matrix whose columns are orthogonal. Notice that Ũ Ũ = I D m but ŨŨ I D! The Moore-Penrose pseudoinverse : A = ŨDiag[λ λ 1 D m ]Ũ

8 The Choleski Factorization We consider the case of positive definite symmetric matrices. They can be written as A = BB but the choice of B is not unique. Any such matrix can be decomposed as: A = LL with L being a low-triangular matrix with nonnegative diagonal entries. This decomposition is unique. Complexity of Choleski decomposition algorithms for a D D non singular matrix: D 3 FLOPS. This is twice more efficient than the LU decomposition. Let Ax = b. No matrix inversion needed to solve it! This can be rewritten as: { Ly = b L x = y

9 Matrix Norms The Frobenius norm: A 2 F = tr(a A) = tr(uλ 2 U ) = tr(λ 2 ) = D i=1 λ 2 i The spectral norm: max v ( Av v = max v (see the Raylegh-Ritz theorem below) v A ) 1/2 Av v = λ max v

10 Geometric Interpretation Consider a positive definite symmetric matrix; In this case all the eigenvalues are strictly positive. Quadratic form for any vector x 0: x Ax = (U x) Λ(U x) = D λ i (u i x) 2 i=1 Let s transform the data into another coordinate frame: z = U x; we obtain: x Ax = z Λz. z Λz = (z 1 /λ 1/2 1 ) (z D /λ 1/2 )2 = C This is an ellipsoid with axes u 1... u D and with half eccentricities λ 1/ λ 1/2 D (Remember PCA...) D

11 The Raylegh-Ritz Theorem Theorem (Raylegh-Ritz). Let A be a symmetric matrix with ordered eigenvalues, then: λ 1 x x x Ax λ D x x x x Ax λ max = λ D = max x 0 x x = max x Ax x x=1 x Ax λ min = λ 1 = min x 0 x x = x min x Ax x=1

12 Proof of the Raylegh-Ritz Theorem From the eigendecomposition: x Ax = D i=1 λ ( i (U ) 2 x) i Notice that: D ( i=1 (U ) 2 x) i = U x 2 = x 2 = x x Using the fact that the eigenvalues can be ordered, we get the first part of the theorm. By dividing we obtain: λ min x Ax x x λ max, (x 0) with equalities when x is a λ 1 or λ D eigenvector. We have: x Ax x x = (x / (x x))a(x/ (x x)) and hence the minimization/maximization of the Raleigh quotient is equivalent to: { max x x Ax x x = 1

13 What About the Remaining Eigenvalues and Eigenvectors? Let s restrict x to be orthogonal to the smallest eigenvector u 1, i.e, u 1 x = 0: x Ax = D i=2 λ i ( (U x) i ) 2 λ2 x x with equality when x = u 2 Therefore we obtain: λ 2 = min x x = 1 x u 1 = 0 x Ax λ D 1 = max x x = 1 x u D = 0 x Ax

14 Computing Eigenvalues and Eigenvectors in Practice The power method estimates the largest eigenvalue/eigenvector pair or an eigenpair. The power method + deflation estimates the second largest eigenpair, etc. The inverse power method estimates the smallest eigenpair. The shifted inverse power method allows to obtain intermediate eigenpairs. The Lanczos method is an adaptation of the power method. It is very useful for large and sparse matrices. It is used by the ARPACK package.

15 The Power Method Input: A symmetric matrix A and a random vector x 0. At each iteration k: 1 Normalize y k = x k x k and 2 x k+1 = Ay k. Check for convergence: y k+1 y k < ε Output: u D = y k+1 and λ D = y k+1 Ay k+1

16 Justification of the Power Method Let x 0 = D i=1 α iu i hence we obtain after the first iteration: x 1 = Ax 0 = D i=1 α iλ i u i Normalize this vector: y 1 = 1 β 1 D i=1 α iλ i u i More generally: y k+1 = 1 D β 1...β k+1 i=1 α iλ k+1 i u i At the limit this vector becomes the largest eigenvector: y = ( α D λ k+1 D 1 ) D α i λ k+1 i lim k β 1... β k+1 α i=1 D λ k+1 u i + u D = u D D λ D = y Ay

17 The Power Method with Deflation Consider the matrix à = A λ Du D u D Notice that (0, u D ) is an eigenpair of à and that the remaining eigenpairs remain unchanged (refer to the spectral decomposition of A and to the fact that eigenvectors corresponding to distinct eigenvalues are orthogonal). It follows that the second largest eigenpair (λ D 1, u D 1 ) of A becomes the largest eigenpair of à The power method can now be applied to Ã, etc.

18 The Inverse Power Method The smallest eigenvector-eigenvalue pair (u 1, λ 1 ) of A corresponds to the largest eigenvector-eigenvalue pair (u 1, λ 1 1 ) of A 1. The k-th iteration of the power method becomes: x k+1 = A 1 y k which can be written as: Ax k+1 = y k This can be solved using the Choleski factorization A = LL : { Lz = yk L x k+1 = z

19 The Shifted Inverse Power Method Let s consider the matrix B = A αi as well as an eigenpair Au = λu. (λ α, u) becomes an eigenpair of B, indeed: Bu = (A αi)u = (λ α)u and hence B is a real symmetric matrix with eigenpairs (λ 1 α, u 1 ),... (λ i α, u i ),... (λ D α, u D ) If α > 0 is choosen such that λ j α λ i α i j then λ j α becomes the smallest (in magnitude) eivenvalue. The inverse power method (in conjuction with the LU decomposition of B) can be used to estimate the eigenpair (λ j α, u j ).