Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Doug James (and Justin Solomon) CS 205A: Mathematical Methods Eigenproblems II: Computation 1 / 31
Setup A R n n symmetric x 1,..., x n R n eigenvectors λ 1 λ 2 λ n eigenvalues CS 205A: Mathematical Methods Eigenproblems II: Computation 2 / 31
Setup A R n n symmetric x 1,..., x n R n eigenvectors λ 1 λ 2 λ n eigenvalues Review (Spectral Theorem): What do we know about the eigenvectors? CS 205A: Mathematical Methods Eigenproblems II: Computation 2 / 31
Usual Trick v R n v = c 1 x 1 + + c n x n CS 205A: Mathematical Methods Eigenproblems II: Computation 3 / 31
Observation λ k 1 ( A k v = ( ) k ( ) ) k c 1 x 1 + λ2 λ λ c2 1 x 2 + + n λ cn 1 x n CS 205A: Mathematical Methods Eigenproblems II: Computation 4 / 31
For Large k A k v λ k 1c 1 x 1 (assuming λ 2 < λ 1 and c 1 0) CS 205A: Mathematical Methods Eigenproblems II: Computation 5 / 31
Power Iteration v k = A v k 1 CS 205A: Mathematical Methods Eigenproblems II: Computation 6 / 31
Power Iteration v k = A v k 1 Question: What if λ 1 > 1? CS 205A: Mathematical Methods Eigenproblems II: Computation 6 / 31
Normalized Power Iteration w k = A v k 1 v k = w k w k CS 205A: Mathematical Methods Eigenproblems II: Computation 7 / 31
Normalized Power Iteration w k = A v k 1 v k = w k w k Question: Which norm? CS 205A: Mathematical Methods Eigenproblems II: Computation 7 / 31
Eigenvalues of Inverse Matrix A v = λ v = A 1 v = 1 λ v CS 205A: Mathematical Methods Eigenproblems II: Computation 8 / 31
Eigenvalues of Inverse Matrix A v = λ v = A 1 v = 1 λ v Question: What is the largest-magnitude eigenvalue? CS 205A: Mathematical Methods Eigenproblems II: Computation 8 / 31
Inverse Iteration w k = A 1 v k 1 v k = w k w k CS 205A: Mathematical Methods Eigenproblems II: Computation 9 / 31
Inverse Iteration w k = A 1 v k 1 v k = w k w k Question: How to make faster? CS 205A: Mathematical Methods Eigenproblems II: Computation 9 / 31
Inverse Iteration with LU Solve L y k = v k 1 Solve U w k = y k Normalize v k = w k w k CS 205A: Mathematical Methods Eigenproblems II: Computation 10 / 31
Eigenvalues of Shifted Matrix A v = λ v = (A σi) v = (λ σ) v CS 205A: Mathematical Methods Eigenproblems II: Computation 11 / 31
Shifted Inverse Iteration To find eigenvalue closest to σ: v k+1 = (A σi) 1 v k (A σi) 1 v k CS 205A: Mathematical Methods Eigenproblems II: Computation 12 / 31
Heuristic: Convergence Rate Recall power iteration: λ k 1 ( A k v = ( ) k ( ) ) k c 1 x 1 + λ2 λ λ c2 1 x 2 + + n λ cn 1 x n CS 205A: Mathematical Methods Eigenproblems II: Computation 13 / 31
Strategy for Better Convergence For power iteration, find σ with λ 2 σ λ 1 σ < λ 2 λ 1 CS 205A: Mathematical Methods Eigenproblems II: Computation 14 / 31
Least-Squares Approximation If v 0 is approximately an eigenvector: arg min λ A v 0 λ v 0 2 2 = v 0 A v 0 v 0 2 2 CS 205A: Mathematical Methods Eigenproblems II: Computation 15 / 31
Rayleigh Quotient Iteration w k = (A σ k I) 1 v k 1 v k = w k w k σ k+1 = v k A v k v k 2 2 CS 205A: Mathematical Methods Eigenproblems II: Computation 16 / 31
Rayleigh Quotient Iteration w k = (A σ k I) 1 v k 1 v k = w k w k σ k+1 = v k A v k v k 2 2 Efficiency per iteration vs. number of iterations? CS 205A: Mathematical Methods Eigenproblems II: Computation 16 / 31
Unlikely Failure Mode for Iteration What is v 0? CS 205A: Mathematical Methods Eigenproblems II: Computation 17 / 31
Unlikely Failure Mode for Iteration What is v 0? What happens when v 0 x 1 = 0? CS 205A: Mathematical Methods Eigenproblems II: Computation 17 / 31
Bug or Feature? 1. Compute x 0 via power iteration. 2. Project x 0 out of v 0. 3. Compute x 1 via power iteration. 4. Project span { x 0, x 1 } out of v 0. 5. CS 205A: Mathematical Methods Eigenproblems II: Computation 18 / 31
Bug or Feature? 1. Compute x 0 via power iteration. 2. Project x 0 out of v 0. 3. Compute x 1 via power iteration. 4. Project span { x 0, x 1 } out of v 0. 5. Assumption: A is symmetric. CS 205A: Mathematical Methods Eigenproblems II: Computation 18 / 31
Avoiding Numerical Drift Do power iteration on P AP where P projects out known eigenvectors. Deflation Modify A so that power iteration reveals an eigenvector you have not yet computed. CS 205A: Mathematical Methods Eigenproblems II: Computation 19 / 31
Similarity Transformations Similar matrices Two matrices A and B are similar if there exists T with B = T 1 AT. CS 205A: Mathematical Methods Eigenproblems II: Computation 20 / 31
Similarity Transformations Similar matrices Two matrices A and B are similar if there exists T with B = T 1 AT. Proposition Similar matrices have the same eigenvalues. CS 205A: Mathematical Methods Eigenproblems II: Computation 20 / 31
Householder Asymmetric Deflation H x 1 = e 1 = HAH e 1 = HAH e 1 by symmetry = HA x 1 since H 2 = I = λ 1 H x 1 = λ 1 e 1 CS 205A: Mathematical Methods Eigenproblems II: Computation 21 / 31
Householder Asymmetric Deflation HAH = ( λ 1 b 0 B ) Similarity transform of A = same eigenvalues. CS 205A: Mathematical Methods Eigenproblems II: Computation 22 / 31
Householder Asymmetric Deflation HAH = ( λ 1 b 0 B ) Similarity transform of A = same eigenvalues. Do power iteration on B. CS 205A: Mathematical Methods Eigenproblems II: Computation 22 / 31
Householder Asymmetric Deflation HAH = ( λ 1 b 0 B ) Similarity transform of A = same eigenvalues. Do power iteration on B. Reveals eigenvalues + vectors one at a time. CS 205A: Mathematical Methods Eigenproblems II: Computation 22 / 31
<warning> Justin s favorite algorithm. Ever. </warning> CS 205A: Mathematical Methods Eigenproblems II: Computation 23 / 31
Conjugation without Inversion Q 1 = Q = Q 1 AQ = Q AQ CS 205A: Mathematical Methods Eigenproblems II: Computation 24 / 31
Conjugation without Inversion Q 1 = Q = Q 1 AQ = Q AQ But which Q? Should involve matrix structure but be easy to compute. CS 205A: Mathematical Methods Eigenproblems II: Computation 24 / 31
Experiment A = QR Q 1 AQ =? CS 205A: Mathematical Methods Eigenproblems II: Computation 25 / 31
QR Iteration A 1 = A Factor A k = Q k R k Multiply A k+1 = R k Q k CS 205A: Mathematical Methods Eigenproblems II: Computation 26 / 31
Commutativity Lemma Take A, B R n n. Suppose that the eigenvectors of A span R n and have distinct eigenvalues. Then, AB = BA if and only if A and B have the same set of eigenvectors (with possibly different eigenvalues). CS 205A: Mathematical Methods Eigenproblems II: Computation 27 / 31
If QR Iteration Converges A = Q R = R Q (Convergence proof in book.) CS 205A: Mathematical Methods Eigenproblems II: Computation 28 / 31
Starting Point (A + δa)( x + δ x) = (λ + δλ)( x + δ x) CS 205A: Mathematical Methods Eigenproblems II: Computation 29 / 31
Starting Point (A + δa)( x + δ x) = (λ + δλ)( x + δ x) What are the independent and dependent variables? CS 205A: Mathematical Methods Eigenproblems II: Computation 29 / 31
Starting Point (A + δa)( x + δ x) = (λ + δλ)( x + δ x) What are the independent and dependent variables? Approximation: Aδ x + δa x λδ x + δλ x CS 205A: Mathematical Methods Eigenproblems II: Computation 29 / 31
Trick: Left Eigenvector A x = λ x, x 0 = y 0 such that A y = λ y CS 205A: Mathematical Methods Eigenproblems II: Computation 30 / 31
Change in Eigenvalue δλ δa 2 y x CS 205A: Mathematical Methods Eigenproblems II: Computation 31 / 31
Change in Eigenvalue δλ δa 2 y x What about symmetric A? Next CS 205A: Mathematical Methods Eigenproblems II: Computation 31 / 31