AMS526: Numerical Analysis I (Numerical Linear Algebra)

Transcription

1 AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 16: Rayleigh Quotient Iteration Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 10

2 Solving Eigenvalue Problems All eigenvalue solvers must be iterative Iterative algorithms have multiple facets: 1 Basic idea behind the algorithms 2 Convergence and techniques to speed-up convergence 3 Efficiency of implementation 4 Termination criteria We will focus on first two aspects Xiangmin Jiao Numerical Analysis I 2 / 10

3 Simplification: Real Symmetric Matrices We will consider eigenvalue problems for real symmetric matrices, i.e. A = A T R m m, and Ax = λx for x R m Note: x = x T, and x = x T x A has real eigenvalues λ 1,λ 2,..., λ m and orthonormal eigenvectors q 1, q 2,..., q m, where q j = 1 Eigenvalues are often also ordered in a particular way (e.g., ordered from large to small in magnitude) In addition, we focus on symmetric tridiagonal form Why? Because phase 1 of two-phase algorithm reduces matrix into tridiagonal form Xiangmin Jiao Numerical Analysis I 3 / 10

4 Rayleigh Quotient The Rayleigh quotient of x R m is the scalar r(x) = x T Ax x T x For an eigenvector x, its Rayleigh quotient is r(x) = x T λx/x T x = λ, the corresponding eigenvalue of x For general x, r(x) = α that minimizes Ax αx 2. x is eigenvector of A r(x) = x 2 T x (Ax r(x)x) = 0 with x 0 r(x) is smooth and r(q j ) = 0 for any j, and therefore is quadratically accurate: r(x) r(q J ) = O( x q J 2 ) as x q J for some J Xiangmin Jiao Numerical Analysis I 4 / 10

5 Power Iteration Simple power iteration for largest eigenvalue Algorithm: Power Iteration v (0) =some unit-length vector for k = 1, 2,... w = Av (k 1) v (k) = w/ w λ (k) = r(v (k) ) = (v (k) ) T Av (k) Termination condition is omitted for simplicity Xiangmin Jiao Numerical Analysis I 5 / 10

6 Convergence of Power Iteration Expand initial v (0) in orthonormal eigenvectors q i, and apply A k : v (0) = a 1 q 1 + a 2 q a m q m v (k) = c k A k v (0) = c k (a 1 λ k 1q 1 + a 2 λ k 2q a m λ k mq m ) = c k λ k 1(a 1 q 1 + a 2 (λ 2 /λ 1 ) k q a m (λ m /λ 1 ) k q m ) If λ 1 > λ 2 λ m 0 and q T 1 v (0) 0, this gives ( v (k) (±q 1 ) = O λ 2 /λ 1 k), λ (k) λ 1 = O ( λ 2 /λ 1 2k) where ± sign is chosen to be sign of q T 1 v (k) It finds the largest eigenvalue (unless eigenvector is orthogonal to v (0) ) Error reduces by only a constant factor ( λ 2 /λ 1 ) each step, and very slowly especially when λ 2 λ 1 Xiangmin Jiao Numerical Analysis I 6 / 10

7 Inverse Iteration Apply power iteration on (A µi ) 1, with eigenvalues {(λ j µ) 1 } If µ λ J for some J, then (λ J µ) 1 may be far larger than (λ j µ) 1, j J, so power iteration may converge rapidly Algorithm: Inverse Iteration v (0) =some unit-length vector for k = 1, 2,... Solve (A µi )w = v (k 1) for w v (k) = w/ w λ (k) = r(v (k) ) = (v (k) ) T Av (k) Converges to eigenvector q J if parameter µ is close to λ J ( v (k) µ λ J k) ( (±q J ) = O, λ (k) µ λ J λ J = O µ λ K µ λ K where λ J and λ K are closest and second closest eigenvalues to µ Standard method for determining eigenvector given eigenvalue 2k ) Xiangmin Jiao Numerical Analysis I 7 / 10

8 Rayleigh Quotient Iteration Parameter µ is constant in inverse iteration, but convergence is better for µ close to the eigenvalue Improvement: At each iteration, set µ to last computed Rayleigh quotient Algorithm: Rayleigh Quotient Iteration v (0) =some unit-length vector λ (0) = r(v (0) ) = (v (0) ) T Av (0) for k = 1, 2,... Solve (A λ (k 1) I )w = v (k 1) for w v (k) = w/ w λ (k) = r(v (k) ) = (v (k) ) T Av (k) Cost per iteration is linear for tridiagonal matrix Xiangmin Jiao Numerical Analysis I 8 / 10

9 Convergence of Rayleigh Quotient Iteration Cubic convergence in Rayleigh quotient iteration v (k+1) (±q J ) = O( v (k) (±q J ) 3 ) and λ (k+1) λ J = O ( λ (k) λ J 3) In other words, each iteration triples number of digits of accuracy Proof idea: If v (k) is close to an eigenvector, v (k) (±q J ) ɛ, then accuracy of Rayleigh quotient estimate λ (k) is λ (k) λ J = O(ɛ 2 ). One step of inverse iteration then gives v (k+1) q J = O( λ (k) λ J v (k) q J ) = O(ɛ 3 ) Rayleigh quotient is great in finding largest (or smallest) eigenvalue and its corresponding eigenvector. What if we want to find all eigenvalues? Xiangmin Jiao Numerical Analysis I 9 / 10

10 Operation Counts In Rayleigh quotient iteration, if A R m m is full matrix, then solving (A µi )w = v (k 1) may take O(m 3 ) flops per step if A R m m is upper Hessenberg, then each step takes O(m 2 ) flops if A R m m is tridiagonal, then each step takes O(m) flops Xiangmin Jiao Numerical Analysis I 10 / 10