Iterative Methods for Eigenvalues of Symmetric Matrices as Fixed Point Theorems

Transcription

1 Iterative Methods for Eigenvalues of Symmetric Matrices as Fixed Point Theorems Student: Amanda Schaeffer Sponsor: Wilfred M. Greenlee December 6, 007. The Power Method and the Contraction Mapping Theorem Understanding the following derivations, definitions, and theorems may be helpful to the reader. The Power Method Let A be a symmetric n x n matrix with eigenvalues > λ λ 3 λ 4... λ n 0 and corresponding orthonormal eigenvectors u, u, u 3,..., u n. We call the dominant eigenvalue, and u the dominant eigenvector. The Power Method is a basic method of iteration for computing this dominant eigenvector. The algorithm is as follows:. Choose a unit vector, x 0.. Let y = A x Normalize, letting x = y y. 4. Repeat steps and 3, letting y k = A x k, until x k converges.

2 [4] The following shows that the limit of this sequence is the dominant eigenvector, provided that ( x u ) is nonzero. Let x R n. Because the eigenvectors of A are orthonormal and therefore span R n, we can write: x = n i= (α i u i ), where α i R. Then multiplying by the matrix, A, we find: By factoring λ k, we obtain: A x = n i= (α i A u i ) = n i= (α i λ i u i ) A x = n i= (α i Aλ i u i ) = n i= (α i λ i u i )...A k x = n i= (α i λ k i u i ). A k x = λ k ni= α i ( λ i ) k u i = λ k α u + λ k ni= α i ( λ i ) k u i. So since λ λ i for all i =, 3,..., n, we have that A k x λ k = α u + O( λ ) k as k. Definition: Metric Space A set of elements, X, is said to be a metric space if to each pair of elements u, v X, there is associated a real number d(u, v), the distance between u and v, such that:. d(u, v) > 0 for u, v distinct,. d(u, u) = 0, 3. d(u, v) = d(v, u), 4. d(u, w) d(u, v) + d(v, w) (The Triangle Inequality holds). [5]

3 Definition: Complete Metric Space Let X be a metric space. X is said to be complete if every Cauchy sequence in X converges to a point of X. Definition: Contraction Let X be a metric space. A transformation, T : X X, is called a contraction if for some fixed ρ <, d(t u, T v) ρd(u, v) for all u, v X. [5] The Contraction Mapping Theorem Let T be a contraction on a complete metric space X. Then there exists exactly one solution, u X, to u = T u. [5] Note: We leave the proof of this theorem to the discussion of our specific example below.. A contraction for finding the dominant eigenvector Let A be a symmetric n x n matrix with eigenvalues > λ λ 3 λ 4... λ n 0 and corresponding orthonormal eigenvectors u, u, u 3,..., u n. Let x, y R n. Because the eigenvectors of A are orthonormal and therefore span R n, we can write: x = n i= (α i u i ) and y = n i= (β i u i ) with α i, β i R. These can also be defined as: α i = ( x u i ), β i = ( y u i ). The Contraction Define the transformation T = A. Then we have the following: T x = α u + i> α i ( λ i ) u i T y = β u + i> β i ( λ i ) u i T x T y = (α β ) u + i>(α i β i )( λ i ) u i. 3

4 Now, consider T x T y. In order to have a contraction, we want some 0 < k < such that T x T y < k x y. To find this, we will use T x T y. Then we want our k such that T x T y < k x y. Since the u i s are orthonormal, we find that In the case that β = α, T x T y = (α β ) + i>((α i β i ) ( λ i ) ). T x T y = i>((α i β i ) ( λ i ) ) ( λ ) i>((α i β i ) ) = ( λ ) x y so, T x T y ( λ ) x y. Then T x T y λ x y. Therefore, when β = α, we have T as a contraction with k = λ. The Fixed Point Let x 0 R n be our initial guess for finding an eigenvector. Define: V = { y R n ( y u ) = ( x 0 u )}. (ie, V is the set of vectors such that β = α ). Then T maps V back into V. Because this set is a closed subset of R n, and R n is a complete metric space, V is also complete. Then by the Contraction Mapping Theorem, there is exactly one y f V such that y f = T y f. It follows (from the below proof) that y f = ( x 0 u ) u is the unique fixed point. We now prove the Contraction Mapping Theorem in our context, as mentioned in Part. Proof: (a) Uniqueness : The first thing we need to show is that the fixed point of our contraction is unique. Suppose u = T u and v = T v. Then v u = 4

5 T v T u. But then, T v T u λ v u. So by substitution, v u λ v u. But then v u = 0, so v = u. Thus, there is only one fixed point, y f. (b) Existence : Next, we want to show that there exists a fixed point at all. Again, let x 0 be our initial guess. Define x m = T x m. Then we have: x m x m+ = T x m T x m λ x m x m... λ m x 0 x. Therefore, for n > m, we see that x m x n x m x m x n x n λ m x 0 x λ n x 0 x = x 0 x λ m ( + λ + λ λ n m ) x 0 x λ m. λ Note that since 0 < λ <, we replaced the geometric sum with the sum of the infinite series. Now we see that {x k } forms a Cauchy sequence, and thus converges to an element of V since V is complete. Call the limit x. Then T is a contraction, and hence continuous, so x is the fixed point, since: x = lim n x n = lim n T x n = T (lim n x n ) = T x. (c) The fixed point x is y f = ( x 0 u ) u : Finally, we want to find the fixed point for our particular contraction, T = A. Consider T ( x 0 u ) u. This is A ( x 0 u ) u = ( x 0 u ) u. Then this is the fixed point. The Schwartz Quotient Because we do not know, we must find an adequate approximation. For this, we use the Schwartz quotient. Consider a positive symmetric matrix A. We define the Schwartz quotient, Λ (m), as follows: Λ (m) = ( Am+ x 0 A m x 0 A m x 0 A m x 0 ) We show that this converges to, provided α = ( x 0 u ) 0. 5

6 since λ i < for each i =,..., n Λ (m) = ( Am+ x 0 A m x 0 A m x 0 A m x 0 ) α = λ + n i= ( λ i ) λ m+ α i α + n i= ( λ i ) λ m α i, But also, Λ (m) α = λ + n i= ( λ i ) λ m+ α i α + n i= ( λ i ) λ m α i α λ α + n i= ( λ i ) λ m α i α λ α +( λ ) λ m n i= α i α = λ α +( λ ) λ m ( α ) (Notice: α + n i= α i = n i= α i = x 0 =, since x 0 is normalized.) So we find that and thus, α α +( λ ) m ( α ) Λ(m), Λ (m) = + O( λ ) m. Notice that for a negative symmetric matrix, the inequalities are reversed, though the end result remains unchanged. Also note that the asymptotic conclusion Λ (m) = + O( λ ) m holds without the assumption that A is definite. This follows by use of a finite geometric series argument based on the explicit formula Λ (m) α = λ + n i= ( λ i ) λ m+ α i α + n. i= ( λ i ) λ m α i 6

7 Then we define a new set of transformations S m : V V such that S m = A Λ (m). By applying S m to A and normalizing, we iterate to find y f, a procedure which converges to the dominant eigenvector, u, as in the case with the original mapping. Note: There are two manners of doing this iteration. In the way we analyze below, call this Version I, we iterate the Schwartz quotient until it is satisfactorily close to the dominant eigenvalue, and then we iterate using the corresponding S m to find the dominant eigenvector. However, in the example below, we use what we will call Version II of the iteration. We found the corresponding S m for each iteration of the Schwartz quotient, and thus with each iteration, the transformation was different. While this version of the iteration still converges to the dominant eigenvector, finding the rate of convergence is more difficult than that of the first version due to the nature of this type of iteration. In the following sections, we will find the rate of convergence of Version II. Then Recall: T x 0 = A ( x 0 ) = α u + i> α i ( λ i ) u i Now we find that, using Version I: S m ( x 0 ) = A λ ( x Λ (m) 0 ) = α u Λ (m) + n i= ( λ i )α Λ (m) i u i T m ( x 0 ) Sm( m x 0 ) = α ( ) m u Λ (m) + i> α i [( λ i ) m ( λ i ) m ] u Λ (m) i T m ( x 0 ) S m m( x 0 ) = α ( Λ (m) ) m + i> α i [( λ i ) m ( λ i Λ (m) ) m ] = α ( Λ (m) ) m + i> α i ( λ i ) m [ ( Λ (m) ) m ] But, recall: So T m ( x 0 ) S m m( x 0 ) = O( Λ (m) ) m Λ (m) = + O( λ ) m 7

8 Now, T m ( x 0 ) Sm( m λ x 0 ) = O( +O( λ )m ) m = O( λ ) m S m m( x 0 ) α u = S m m( x 0 ) T m ( x 0 ) + T m ( x 0 ) α u S m m( x 0 ) T m ( x 0 ) + T m ( x 0 ) α u by the triangle inequality. So, = O( λ ) m + O( λ ) m S m m( x 0 ) α u = O( λ ) m = T m ( x 0 ) α u. Then these transformations have the same rate of convergence. 3. Example ) Consider the following 3 3 matrix (found in [3], Ex. 4--): This matrix has known eigenvalues and corresponding eigenvectors ; λ 9.470; λ , u , 0.679, 0.3 ; u , , ; u , ,

9 Using the Schwartz Quotient to find the approximate eigenvalue, Λ (m), and Version II of our fixed point mapping of the power method to find the approximate eigenvector, v m, and beginning with an initial guess x 0 =,,, we find: m Λ (m) v m ,., , , , , , , , , , , , 0.68, , , , , , , , 0.676, Monotonicity of Schwartz Quotients In this section, we alter a proof found in Collatz [] to find that in the two cases of a symmetric, positive definite matrix or negative definite matrix, the sequence of Schwartz Quotients, {Λ (m) }, where Λ (m) = ( Am+ x 0 A m x 0 A m x 0 A m x 0 ), is monotone increasing or decreasing, respectively. This is also the behavior observed in the preceding example, with an indefinite matrix. Case : The positive definite symmetric matrix Consider a matrix, A, which is positive definite. The m-th term in the sequence of Schwartz Quotients is Λ (m) = ( Am+ x 0 A m x 0 A m x 0 A m x 0 ). Define a m+ = A m+ x 0 A m x 0 and a m = A m x 0 A m x 0. Then Λ (m) = a m+ a m. Note: In a similar manner, we define a m as (A m x 0 A m x 0 ), etc. 9

10 Then, let Q = a m+ A m x 0 a m A m+ x 0. This value is obviously nonnegative. But then Q = (a m+ A m x 0 a m A m+ x 0 ) (a m+ A m x 0 a m A m+ x 0 ) By the bilinearity of the inner product, this is Then and hence = (a m+ A m x 0 ) (a m+ A m x 0 ) (a m+ A m x 0 ) (a m A m+ x 0 ) (a m A m+ x 0 ) (a m+ A m x 0 ) + (a m A m+ x 0 ) (a m A m+ x 0 ) = (a m+a m ) (a m+ a m a m+ ) + (a ma m+ ) = (a ma m+ ) (a m+a m ) Using this result, we find that (a ma m+ ) (a m+a m ) 0, (a ma m+ ) (a m+a m ). Q (a ma m+ ) = (a m+ a m+ a m+ a m ) 0 () since we assume A positive definite, and thus a m+ > 0. Now define Q = (a m+ A m x 0 a m A m+ x 0 ) (a m+ A m x 0 a m A m x 0 ). Since A is positive, we know Q is non-negative. But then, So, consider now and thus Q = a m+a m a ma m+ + a ma m+ = a m+ (a m+ a m a m) Q a m a m a m+. This gives us: a m+ a m a m a m 0, a m+ a m a m a m () Then by () and (), we see that Λ (m) Λ (m+). So we have shown that for a positive definite matrix A, the sequence of Schwartz quotients is monotone 0

11 increasing. Case : The negative definite symmetric matrix An analogous result can easily be seen for a negative definite matrix, where the in () and () are replaced with since A is negative. Thus we find that in this case, the sequence of Schwartz quotients is monotone decreasing. Case 3: The indefinite symmetric matrix Unfortunately, we have yet to show a similar result for the case of the indefinite matrix. However, we know that the Schwartz quotients still converge to the dominant eigenvalue, and the mapping converges in a similar way as in the first two cases, which we now show for version II. 5. Convergence of the Fixed Point Mapping, Version II Suppose A is a definite matrix and x 0 is our initial guess. Note that, using Version II of our fixed point mapping, S x 0 = A Λ () ( x 0 ) = α Λ () u + n i= ( λ i Λ () )α i u i. If we let this = x, then in the next iteration, x = S x = A λ ( x Λ () ) = α u Λ () Λ () + n i= ( )α Λ () Λ () i u i. Continuing in this manner, we arrive at the m-th iteration, λ i Now, λ S m x m = α m u Λ () Λ ()...Λ (m) + n i= ( )α Λ () Λ ()...Λ (m) i u i λ m i λ m i n i= ( )α Λ () Λ ()...Λ (m) i u i λ m i = n i= ( )α (Λ () Λ ()...Λ (m) ) i

12 = ni= λ m (Λ () Λ ()...Λ (m) ) i αi λ m (Λ () Λ ()...Λ (m) ) ( α ) Now, let 0 < k < be such that k > λ. Because Λ (m) converges to, there exists some index, M, such that for all m M, Then for m > M, Since ( λ m Λ (m) > k > λ. λ M = ( )( (Λ () Λ ()...Λ (m) ) (Λ () Λ ()...Λ (M) ) λ M ( )( λ (Λ () Λ ()...Λ (M) ) k ) m M = ( λ M )( kλ (Λ () Λ ()...Λ (M) ) λ m M ) (Λ (M+)...Λ (m) ) λ M )( kλ (Λ () Λ ()...Λ (M) ) = O( λ k ) m as m. λ ) M is independent of m > M. λ ) M ( λ k ) m So we see that S m x m approaches a multiple of u, as does the usual power method. 6. Applying Our Contraction to the Shifted Inverse Power Method The Shifted Inverse Power Method The Shifted Inverse Power Method is an alteration to the Power Method which finds the eigenvalue of a matrix closest to a chosen value rather than finding the dominant eigenvalue. The basic idea is to choose a value, q, and rather than using the original matrix, A, find (A qi) and apply the Power Method to this shifted inverse matrix. The matrix (A qi) has eigenvalues,,..., q λ q λ n q and corresponding eigenvectors u,..., u n, where the λ i s are again the eigenvalues of A, > λ λ 3 λ 4... λ n > 0, and the u i s are the corresponding

13 orthonormal eigenvectors to the λ i s in A. As in the original case, the Power Method will approximate the dominant eigenvalue of (A qi), which is, where λ λ j q j is the closest eigenvalue of A to q. Our Contraction The mapping used in this case is completely analogous to in the case of the original Power Method. We define T inv by: T inv x = (λ j q)(a qi) x, where again λ j is the closest eigenvalue of A to q. For any x, we again have that x = n i= (α i u i ) since the eigenvectors span R n. Then we have that: Taking the m-th iteration, T inv x = (λ j q) n i= ( )(α λ i q i u i ) = ( λ j q )α λ j q j u j + i j( λ j q )(α λ i q i u i ) = α j u j + i j( λ j q )(α λ i q i u i ) T m inv x = α j u j + i j( λ j q λ i q )m (α i u i ) Since λ j q < λ i q for all i j, we have that T m inv α j u j as m. Notice that if we let (A qi) = B, then T inv = B γ, where γ is the dominant eigenvalue of B. Then T inv is equivalent to the original mapping, T, for the matrix B. Clearly, then, we can take the Schwartz quotients with respect to B = (A qi), approximating the eigenvalues, Call the m-th Schwartz λ i q. quotient Λ (m) B. Now, with the m-th iteration, we make T inv = B. This is Λ (m) B completely analogous to the mapping for the original power method, and thus has convergence O( λ k q λ j q A to q and λ k is the next closest. ) m = O( λ j q λ k q )m where λ j is the closest eigenvalue of 3

14 References:. L. Collatz, Eigenwertprobleme und ihre Numerische Behandlung, Chelsea, New York, NY, N. J. Higham, Review of Spectra and Pseudospectra: The Behavior of Nonnomral Matrices and Operators, by L. N. Trefethen and M. Embree, Princeton University Press, Princeton, NJ, 005; in Bull AMS (44), 77-84, A. N. Langville and C.D. Meyer, Google s Page Rank and Beyond, Princeton University Press, Princeton, NJ, B.N. Parlett, The Symmetric Eigenvalue Problem, Prentice Hall, Englewood Cliffs, NJ, I. Stakgold, Green s Functions and Boundary Value Problems, Wiley- Interscience, New York, NY, G.W. Stewart, Introduction to Matrix Computations, Academic Press, New York, NY,