Invariant Subspace Computation - A Geometric Approach Pierre-Antoine Absil PhD Defense Université de Liège Thursday, 13 February 2003 1
Acknowledgements Advisor: Prof. R. Sepulchre (Université de Liège) Collaborations with: Prof. P. an Dooren (Université Catholique de Louvain) Prof. R. Mahony (Australian National University) 2
Outline Eigenspace computation: definitions. Eigenspace computation: an application. Geometric approach to eigenspace computation. New numerical algorithms: Newton methods. Shifted inverse iterations. Global behaviour and improvements. 3
Topic: Refining estimates of eigenspaces A Data: n-by-n matrix A p-dimensional subspace, approx. of eigenspace. Task: compute better estimate of. Definition of eigenspace : A. + 4
Application: visual positioning Relative positions Raw images 5
Application: visual positioning Relative positions Raw images 6
Application: visual positioning Relative positions Raw images?? 7
Application: visual positioning Relative positions Raw images?? Dim 10000 8
Application: visual positioning Relative positions Raw images Dim 10000 Projection Dim 20 Processed images 9
Application: visual positioning Relative positions Raw images Dim 10000 Projection Dim 20 Processed images 10
Application: visual positioning Relative positions Raw images Dim 10000 Projection Dim 20 Processed images 11
Application: visual positioning Relative positions Raw images Dim 10000 Projection Dim 20 Processed images 12
Application: visual positioning Relative positions Raw images Dim 10000 Projection Dim 20 Processed images 13
Application: visual positioning Learning... Compute covariance matrix and dominant eigenspace... 14
Application: visual positioning Raw image Projection onto eigenspace. 15
Application: visual positioning Raw image Projection onto eigenspace. 16
Outline Eigenspace computation: definitions. Eigenspace computation: an application. Geometric approach to eigenspace computation. New numerical algorithms: Newton methods. Shifted inverse iterations. Global behaviour and improvements. 17
Refining estimates of eigenspaces A + Grass(p, n) + p = dim() = dim() = dim( + ). Grass(p, n) (Grassmann manifold): the set of p-planes of R n. 18
Matrix representations M = [ y 1... y p] = M (M p p invertible) span = [y 1... y p ] R n p y 2 + y 1 Grass(p, n) 19
Iterations on the Grassmann manifold M = M (M p p invertible) R n p + Grass(p, n) 20
Outline Eigenspace computation: definitions. Eigenspace computation: an application. Geometric approach. New numerical algorithms: Newton methods. Shifted inverse iterations. Global behaviour and improvements. 21
Newton method for eigenspace computation M = M F ( ) := Π A, where Π := I ( T ) 1 T. R n p y 2 + Newton method in R n p : y 1 Grass(p, n) F ( ) + DF ( )[ ] = 0 + = + 22
Newton method for eigenspace computation M = M F ( ) := Π A, where Π := I ( T ) 1 T. y 2 R n p Newton method in R n p : + F ( ) + DF ( )[ ] = 0 y 1 Grass(p, n) + = + Result: =, hence + = 0. 23
Newton method Horizontal update M = M F ( ) := Π A, where Π := I ( T ) 1 T. y 1 y 2 R n p + H Grass(p, n) Newton method with constrained update: F ( ) + DF ( )[ ] = 0 T = 0 Overdetermined system! 24
Overdetermined system: Two remedies: Newton method - Relaxations F ( ) + DF ( )[ ] = 0 T = 0 = arg min F ( ) + DF ( )[ ] 2 = 0 T =0 Π (F ( ) + DF ( )[ ]) = 0 T = 0 25
Newton method Riemannian interpretation M = M F ( ) := Π A Π (F ( ) + DF ( )[ ]) = 0 T = 0 y 2 R n p H + = +. + y 1 Grass(p, n) Riemannian Newton method: F ( ) + F ( ) = 0 + = Exp( ) 26
Newton method for eigenspace computation Iteration mapping: Grass(p, n) + Grass(p, n) defined by 1. Pick a basis R n p that spans and solve the equation ΠAΠ ( T ) 1 T A = ΠA (1) under the constraint T = 0, where Π := I ( T ) 1 T. 2. Perform the update + = span( + ). (2) 27
Newton method Related work Previous work: Stewart [Ste73], Dongarra, Moler and Wilkinson [DMW83], Chatelin [Cha84], Demmel [Dem87]. Lösche, Schwetlick and Timmerman [LST98]. Edelman, Arias and Smith [EAS98]. Lundström and Elden [LE02]. New results (PAA, Mahony, Sepulchre [AMS02]): Iteration converges locally quadratically (even when A A T ). Convergence is locally cubic if T. 28
Newton method Cross sections y 2 R n p M = M S Parameterize S: = W + W K. Then ΠA = 0 becomes a quadratic matrix equation + A 22 K KA 11 + A 21 KA 12 K = 0 y 1 Grass(p, n) where A 11 = W T AW, A 12 = W T AW, A 21 = W T AW and A 22 = W T AW. 29
Newton method Adaptive cross section M = M + R n p y 2 + y 1 Grass(p, n) Advantage: i = A T, convergence is cubic, instead of quadratic in the fixed cross section case. 30
Outline Eigenspace computation: definitions. Eigenspace computation: an application. Geometric approach. New numerical algorithms: Newton methods. Shifted inverse iterations. Global behaviour and improvements. 31
Rayleigh Quotient Iteration Case p = 1 mz mv R n v z y + y y = my (A ρ(y)i)z = y ρ(y) := (y T Ay)/(y T y). span(v) span(y + ) span(y) Grass(1, n) Local cubic convergence to eigendirections o = A T. 32
Rayleigh Quotient Iteration From p = 1 to p 1 M Z = M (A ρ(y)i)z = y ρ(y) := (y T Ay)/(y T y). R n p + + Want: p>1?? Well defined on Grass(p, n). Grass(p, n) Reduces to RQI when p = 1. Local cubic convergence. 33
Rayleigh Quotient Iteration Case p 1 M R n p + + Grass(p, n) Classical Rayleigh Quotient Iteration (p = 1): (A ρ(y)i)z = y ρ(y) := (y T Ay)/(y T y). Generalized iteration (p 1): AZ Z( T ) 1 T A =. 34
Rayleigh Quotient Iteration From p = 1 to p 1 M?? Z = M (A ρ(y)i)z = y ρ(y) := (y T A )/(y T y). R n p + + Grass(p, n) Generalized iteration: AZ Z( T ) 1 T A =. Want: Well defined on Grass(p, n). Reduces to RQI when p = 1. Local cubic convergence. 35
Rayleigh Quotient Iteration From p = 1 to p 1 M ZM Z = M (A ρ(y)i)z = y ρ(y) := (y T A )/(y T y). R n p + + Grass(p, n) Generalized iteration: AZ Z( T ) 1 T A =. Want: Well defined on Grass(p, n). Reduces to RQI when p = 1. Local cubic convergence. 36
Rayleigh Quotient Iteration From p = 1 to p 1 M ZM Z = M (A ρ(y)i)z = y ρ(y) := (y T A )/(y T y). R n p + + Grass(p, n) Generalized iteration: AZ Z( T ) 1 T A =. Want: Well defined on Grass(p, n). Reduces to RQI when p = 1. Local cubic convergence. 37
Rayleigh Quotient Iteration From p = 1 to p 1 M ZM Z = M (A ρ(y)i)z = y ρ(y) := (y T A )/(y T y). R n p + + Grass(p, n) Generalized iteration: AZ Z( T ) 1 T A =. Want: Well defined on Grass(p, n). Reduces to RQI when p = 1. Local cubic convergence. 38
Rayleigh Quotient Iteration From p = 1 to p 1 M ZM Z = M (A ρ(y)i)z = y ρ(y) := (y T A )/(y T y). R n p + + Grass(p, n) Generalized iteration: AZ Z( T ) 1 T A =. Want: Well defined on Grass(p, n). Reduces to RQI when p = 1. Local cubic convergence. 39
Rayleigh Quotient Iteration Local Convergence Z AZ Z( T ) 1 T A = K + + K Result: K + c K 3 R n p + Grass(p, n) 40
Rayleigh Quotient Iteration From p = 1 to p 1 M Z = M (A ρ(y)i)z = y ρ(y) := (y T A )/(y T y). R n p + + Grass(p, n) Generalized iteration: AZ Z( T ) 1 T A =. Want: Well defined on Grass(p, n). Reduces to RQI when p = 1. Local cubic convergence. 41
Rayleigh Quotient Iteration The matrix algorithm Iteration mapping: Grass(p, n) + Grass(p, n) defined by 1. Pick a basis R n p that spans. 2. Solve for Z R n p. 3. Define + as the span of Z. AZ Z( T ) 1 T A = (3) 42
Outline Eigenspace computation: definitions. Eigenspace computation: an application. Geometric approach. New numerical algorithms: Newton methods. Shifted inverse iterations. Global behaviour and improvements. 43
Global behaviour of iterative methods 2 1 + Grass(p, n) Study basins of attraction of eigenspaces. 44
Global behaviour for p = 1 When p = 1, Riemannian Newton method Rayleigh Quotient Iteration. 3 3 X k Grass(1, 3) x k 1 2 X k 1 2 Reference: Batterson and Smillie [BS89]. 45
Newton method Global behaviour gamma = 0.2, p = 1, span(x k+1 ) = span Z 2 gamma = 0.01, p = 1, span(x k+1 ) = span Z 2 v 3 v 1 v 2 1 1.8 1 1.99 Basins may deteriorate when eigenvalues are clustered. 46
Newton method Improved global behaviour p = 2, NG with tau parameter 2 p = 2, NG with tau parameter 2 v 3 v 1 v 2 1 1.8 1 1.99 Remedy: modified Newton methods with large basins of attraction. 47
Rayleigh Quotient Iteration Global behaviour (p > 1) Theoretical results:? Experimental results: AZ Z( T ) 1 T A =, + = ZM gamma = 0.2, p = 2, span(x k+1 ) = span Z 2 gamma = 0.01, p = 2, span(x k+1 ) = span Z 2 v 3 v 1 v 2 1 1.8 1 1.99 48
Rayleigh Quotient Iteration Improved global behaviour (p > 1) gamma = 0.2, p = 2, GRQI with maxvar=.5*pi/5 2 gamma = 0.01, p = 2, GRQI with maxvar=.5*pi/5 2 v 3 v 1 v 2 1 1.8 1 1.99 Modified Rayleigh Quotient Iteration with limited steps. Reference: PAA, Sepulchre, an Dooren, Mahony [ASM03]. 49
Refining eigenspace estimates Main Results Newton methods: Practical formulation of the Riemannian Newton method on the Grassmann manifold. Newton method for refining eigenspace estimates. Quadratic convergence in general. Cubic convergence i = A T. Shifted inverse iterations: Grassmannian generalization of the Rayleigh Quotient Iteration, using block shifts. Local cubic convergence to the eigenspaces o = A T. Two-sided version for the nonsymmetric case. Study of the global behaviour of the iterations. Modified methods with large basins of attraction and high rate of convergence. 50
Refining eigenspace estimates Conclusion M = M R n p y 2 + y 1 Grass(p, n) Geometry: iteration on the Grassmann manifold of p-planes in R n. Dynamics: k+1 = ( k ). Discrete dynamical system. Numerical analysis: compute efficiently. 51
References [AMS02] P.-A. Absil, R. Mahony, and R. Sepulchre, Riemannian geometry of Grassmann manifolds with a view on algorithmic computation, submitted to Acta Appl. Math., November 2002. [ASM03] P.-A. Absil, R. Sepulchre, P. an Dooren, and R. Mahony, Cubically convergent iterations for invariant subspace computation, submitted to SIAM J. Matrix Anal. Appl., January 2003. [BS89] [Cha84] S. Batterson and J. Smillie, The dynamics of Rayleigh quotient iteration, SIAM J. Numer. Anal. 26 (1989), no. 3, 624 636. F. Chatelin, Simultaneous Newton s iteration for the 52
eigenproblem, Computing, Suppl. 5 (1984), 67 74. [Dem87] [DMW83] [EAS98] [LE02] J. W. Demmel, Three methods for refining estimates of invariant subspaces, Computing 38 (1987), 43 57. J. J. Dongarra, C. B. Moler, and J. H. Wilkinson, Improving the accuracy of computed eigenvalues and eigenvectors, SIAM J. Numer. Anal. 20 (1983), no. 1, 23 45. A. Edelman, T. A. Arias, and S. T. Smith, The geometry of algorithms with orthogonality constraints, SIAM J. Matrix Anal. Appl. 20 (1998), no. 2, 303 353. E. Lundström and L. Eldén, Adaptive eigenvalue computations using Newton s method on the Grassmann manifold, SIAM J. Matrix Anal. Appl. 23 (2002), no. 3, 819 839. 53
[LST98] [Ste73] R. Lösche, H. Schwetlick, and G. Timmerman, A modified block Newton iteration for approximating an invariant subspace of a symmetric matrix, Linear Algebra Appl. 275-276 (1998), 381 400. G. W. Stewart, Error and perturbation bounds for subspaces associated with certain eigenvalue problems, SIAM Review 15 (1973), no. 4, 727 764. 54
Numerical experiment: AZ Z T A =, T = I + = qf(z). A = 1 0 0 0 2 0 0 0 3 = -1.0762e-001-4.2770e-002 9.9334e-001 3.6714e-002-4.1138e-002 9.9841e-001 = -1.3004e-003-2.9822e-004-9.8317e-001-1.8269e-001 1.8269e-001-9.8317e-001 = -4.4217e-011 2.3683e-009 1.8740e-002-9.9982e-001-9.9982e-001-1.8740e-002 55
AZ Z( T ) 1 T A = Z orthon. 0 2 1 R n p + Grass(p, n) 56
AZ Z( T ) 1 T A = Z Z 0 T A = diag and T = I 2 1 R n p + Grass(p, n) 57