Numerical solution of the eigenvalue problem for Hermitian Toeplitz-like matrices

Transcription

1 TR-CS-9-1 Numerical solution of the eigenvalue problem for Hermitian Toeplitz-like matrices Michael K Ng and William F Trench July 199 Joint Computer Science Technical Report Series Department of Computer Science Faculty of Engineering and Information Technology Computer Sciences Laboratory Research School of Information Sciences and Engineering

2 This technical report series is published jointly by the Department of Computer Science, Faculty of Engineering and Information Technology, and the Computer Sciences Laboratory, Research School of Information Sciences and Engineering, The Australian National University Please direct correspondence regarding this series to: Technical Reports Department of Computer Science Faculty of Engineering and Information Technology The Australian National University Canberra ACT Australia or send to: A list of technical reports, including some abstracts and copies of some full reports may be found at: Recent reports in this series: TR-CS-9-1 Michael K Ng Blind channel identification and the eigenvalue problem of structured matrices July 199 TR-CS-9-1 Michael K Ng Preconditioning of elliptic problems by approximation in the transform domain July 199 TR-CS-9-11 Richard P Brent, Richard E Crandall, and Karl Dilcher Two new factors of Fermat numbers May 199 TR-CS-9-1 Andrew Tridgell, Richard Brent, and Brendan McKay Parallel integer sorting May 199 TR-CS-9-9 M Manzur Murshed and Richard P Brent Constant time algorithms for computing the contour of maximal elements on the Reconfigurable Mesh May 199 TR-CS-9-8 Xun Qu, Jeffrey Xu Yu, and Richard P Brent socket April 199 A mobile TCP

3 NUMERICAL SOLUTION OF THE EIGENVALUE PROBLEM FOR HERMITIAN TOEPLITZ{LIKE MATRICES MICHAEL K NG AND WILLIAM F TRENCH y Abstract An iterative method based on displacement structure is proposed for computing eigenvalues and eigenvectors of a class of Hermitian Toeplitz{like matrices which includes matrices of the form T T where T is arbitrary Toeplitz matrix, Toeplitz{block matrices and block{toeplitz matrices The method obtains a specic individual eigenvalue (ie, the i-th smallest, where i is a specied integer in [1 :::n]) of an n n matrix at a computational cost of O(n ) operations An associated eigenvector is obtained as a byproduct The method is more ecient than general purpose methods such as the QR algorithm for obtaining a small number (compared to n) of eigenvalues Moreover, since the computation of each eigenvalue is independent of the computation of all other eigenvalues, the method is highly parallelizable Numerical results illustrate the eectiveness of the method Key words Toeplitz matrix, displacement structure, Toeplitz{like matrix, eigenvalue, eigenvector, root{nding AMS subject classications 15A18, 15A5, 5F15 ACSys, Computer Sciences Laboratory, Research School of Information Sciences and Engineering, The Australian National University, Canberra, ACT, Australia mng@cslabanueduau y Department of Mathematics, Trinity University, 15 Stadium Drive, San Antonio, Texas, 81, USA wtrench@trinityedu Research supported by National Science Foundation Grant DMS

4 Michael K Ng and William F Trench 1 Introduction In this paper we consider the eigenvalue problem for an n n Hermitian matrix A n which has displacement structure in the sense that (1) A n Z n, Z n A n = G n H T n where Z n is the shift matrix Z n = G n and H n are in C n, and is small compared to n (For discussions of other types of displacement structure see [8, 1, 11]) The smallest integer for which (1) holds with some G n and H n in C n, is called the fz n Z n g-displacement rank of A n we will call it simply the displacement rank of A n Henceforth we will say that a matrix which satises (1) with small compared to n is a Toeplitz{like matrix There are many ecient direct methods that exploit displacement structure to invert Toeplitz{like matrices, or to solve Toeplitz{like systems A n x = b [, 8, 11] There are also preconditioned conjugate gradient methods for solving Toeplitz-like systems with O(n log n) operations [, ] However, numerical solution of the Toeplitz eigenvalue problem has only recently received attention [5, 9, 15, 1] In particular, Cybenko and Van Loan [5] presented a method for using Levinson's algorithm [1] to nd the smallest eigenvalue of an n n Hermitian Toeplitz matrix with O(n ) operations In [15], Trench extended their method and gave an iterative method for computing arbitrary eigenvalues and associated eigenvectors of Hermitian Toeplitz and Toeplitz{plus{Hankel matrices at a cost of O(n ) per eigenvalue The purpose of this paper is to use Trench's method to compute the eigenvalues and eigenvectors of Hermitian Toeplitz{like matrices In xwe propose an algorithm for nding individual eigenvalues of an nn matrix with displacement rank not greater than at a computation cost of O(n ) each In x we give examples of Hermitian matrices with displacement structure (1), along with specic formulas for the associated matrices G n and H n In x we discuss an application to signal processing In x5 we describe the results of numerical experiments with the algorithm The algorithm The following theorem from [1] provides the motivation and the theoretical basis for the method Part of this theorem goes back at least to Wilkinson [1] (For the statement concerning the inertia of A n, I n, see also Browne [1]) Theorem 1 Let A n =[a ij ] n ij=1 be a Hermitian matrix, and dene Let p () =1, A m =[a ij ] m ij=1 1 m n: p m () = det (A m, I m ) 1 m n and q m () = p m() p m () 1 m n:

5 EIGENVALUE PROBLEM FOR HERMITIAN TOEPLITZ{LIKE MATRICES Dene v m = a 1m+1 a m m n, 1: a mm+1 Let S m be the spectrum of A m and S n = [ n m=1 S m If is real let Neg n () be the number (counting multiplicities) of eigenvalues of A n less than For each = S n let w () =and w m () = be the solutions of the systems w 1m () w m () w mm () 5 1 m n, 1 () (A m, I m )w m () =v m 1 m n, 1: Dene () Then y m () = wm () m n: () (A m, I m )y m () =,q m ()e m m n where e m =[ 1] T is the last column of I m Moreover, q m () =a mm,, v m w m() 1 m n q m() =,kw m ()k and Neg n () equals the number of negative quantities in fq 1 ()q ():::q n ()g Finally, if is an eigenvalue of A n, then y n () is an associated eigenvector Theorem 1 provides a way to compute p n ()=p n () and the inertia of A n,i n Therefore, in principle it can be used in conjunction with a root{nding procedure to determine a given eigenvalue i of A n, provided that i is not \too close" to an eigenvalue of one of the principal submatrices A 1 A :::A n of A n This method is not practical for general Hermitian matrices, because in general O(n ) operations are required to solve the systems () for each value of However, Theorem 1 can be useful if A n is structured so that this computational cost is O(n ) In [15], Trench described a computational strategy combining Theorem 1 with bisection and the Pegasus root{nding method for computing individual eigenvalues and eigenvectors of Hermitian Toeplitz matrices with O(n ) operations In [1] he applied the same strategy to Hermitian Toeplitz{plus{Hankel matrices We will now show that a similar approach can be used to compute individual eigenvalues and eigenvectors of Toeplitz{ like matrices

6 Michael K Ng and William F Trench Henceforth G m and H m (1 m n) are the m matrices obtained by dropping rows m +1:::n from G n and H n thus (5) G m = U mn G n and H m = U mn H n where U mn is the m n matrix obtained by dropping the same rows from I m We denote the jth column of G m by thus, g (m) j = g 1j g mj 5 G m =[g (m) 1 g (m) g (m) ]: The following result of Heinig and Rost [8, p11] is crucial to our approach Lemma If A n satises (1) then () A m Z m, Z m A m = G m H T m, v me T m 1 m n, 1: Proof It is easily veried that U mn A n Z n U T mn = A m Z m + v m e T m and U mn Z n A n U T mn = Z m A m : Therefore we can obtain () by multiplying (1) on the left by U mn and on the right by U T mn, and invoking (5) The following algorithm provides an O(n ) method for solving the linear systems () if A n satises (1) with G n H n C n The algorithm is an adaptation of a recursion formula given in [8, p11] for solving systems with Toeplitz{like matrices Algorithm If =S n then q 1 ():::q n () can be computed as follows: q 1 () =a 11, w 1 () = a 1 q 1 () f (1) j () = g 1j q 1 () 1 j and for m n, () (8) and (9) f (m) j () = w m () = f (m) j () q m () =a mm,, v m w m() w m () y m () = wm () (g mj, v m f (m) j ()), y m () 1 j q m () h, f (m) 1 () f (m) () f (m) () i H T m y m():

7 EIGENVALUE PROBLEM FOR HERMITIAN TOEPLITZ{LIKE MATRICES 5 (1) Proof Adding and subtracting Z m on the left side of () yields (A m, I m )Z m, Z m (A m, I m )=G m H T m, v me T m 1 m n, 1: From () and (), for m n, e T m y m() = Z m (A m, I m )y m () = and Z m ()y m () = Therefore, multiplying (1) on the right by y m () yields (A m, I m ) w m () = G m H T m y m()+v m m n: w m () Multiplying by (A m, I m ) and recalling () shows that this is equivalent to where w m () = w m () which we write in terms of its columns as These columns are the solutions of (11) Since, F m ()H T m y m() F m () =(A m, I m ) G m F m () =[f (m) 1 () f (m) () f (m) ()]: (A m, I m )f (m) j () =g (m) j = g (m) j g mj 1 j : (A m, I m )f (m) j () =g (m) j and (A m, I m )y m () =,q m ()e m it follows that the solutions of (11) are given by (8) Examples The following are examples of Hermitian matrices with the kind of displacement structure indicated in (1) (i) A Hermitian Toeplitz matrix T n =[t i,j ] n ij=1 (where t,r = t r ) has displacement rank at most, since T n Z n, Z n T n = 1 t n t 1 5 t1 t n (ii) If T n =[t i,j ] n ij=1 is an arbitrary (not necessarily Hermitian) n n Toeplitz matrix, then A n = Tn T n has displacement rank at most (see [8, p1] and [1]) It can be shown that A n satises (1) with G n = 1 t t n b 1 5 and H n = t,n+1 t 1 b n t,t n c 1 t,n+1,t 1 c n 5 : :

8 Michael K Ng and William F Trench where b i = nx k=1 t k,i+1 t k,n and c j = n X k=1 t k t k,j : P k (iii) A matrix of the form i=1 T n (i) T n (i), where T n (1) :::T n (k) are arbitrary Toeplitz matrices, has displacement rank at most k [] Matrices like this arise in solving the normal equations of Toeplitz least squares problems in signal and image processing [] (iv) A Hermitian Toeplitz{block matrix of the form (1) A n = T m (11) T m (1) T m (1s) T m (1) T m () T m (s) T m (1s) T m (1s) T m (s) T m (ss) T m (s) T m (ss) where n = sm and ft m (ij) g s ij=1 are Toeplitz matrices given by [T m (ij) ] m = kl=1 t(ij) k,l, has displacement rank at most s [8, p1] For example, if s = it can be shown that (1) holds with n =m, G n = 1 t (1) t (1) 1, t (11),m+1,t (1),m+1 t (1) m, t(11),t (1) 1 t () t () 1, t (1),m+1,t (),m+1 t () m, t(1),t () 5 and H n = 5 t (11) t (1), t (11) m t (11),m+1 t (1),m+1, t(11) 1,t (11) 1 t (1) t (), t (1) m t (1),m+1 t (),m+1, t(1) 1,t (1) 1 (v) For an example closely related to (iv) let B n be the block{toeplitz matrix B n = where each block is an s s matrix thus, h C (r) s = C (p,q) s h c (r) ij i m pq=1 i s : ij=1 Now let P n be the n n permutation matrix dened as follows: for k =1 :::m, rows (k, 1)s + 1 through ks of P n are rows k k + m:::k+(s, 1)m of I n Then A n = P n B n Pn T is the Toeplitz{block matrix where A n = h h Tm ij = T (ij) m c (p,q) ij i s ij=1 i m : pq=1 Moreover, if B n is Hermitian then so is A n that is, A n is of the form (1) Finally, if is an eigenvalue and x is an associated eigenvector of A n, then is an eigenvalue and P T n x is an associated eigenvector of B n 5 :

9 EIGENVALUE PROBLEM FOR HERMITIAN TOEPLITZ{LIKE MATRICES An application to signal processing The input fx k g and the output fy k g of a transversal lter of order n are related by y r = n X k= w k x r,k : In signal processing problems it is often necessary to estimate the lter coecients fw w 1 :::w n g given observed values fx 1 x :::x m g and fy 1 y :::y m g of the input and output, where m>n One way to do this is to choose fw w 1 :::w n g so as to minimize (w w 1 :::w n )= mx r=1 y r, n X k= w k x r,k! where it is assumed that x j = if j An elementary argument shows that fw w 1 :::w n g should be chosen so that nx j=1 a ij w j = mx r=1 y r x r,i+1 1 i n where a ij = mx r=1 x r,i+1 x r,j+1 : The matrix A n =[a ij ] n ij=1 is given by A n = X T X, where X is the m n Toeplitz matrix (1) X = x 1 x x 1 x x 1 x n x1 x m xm,n x m x m x m,n+ x m,n+1 The matrix X T X is called the normal equations matrix or the information matrix of the corresponding least squares problem [1, 1] It is an approximation to the the correlation matrix of the input signal data We are interesting in computing the eigenvalues of X T X because, for example, the smallest and the largest eigenvalues of X T X are related to the accuracy of the least squares computations and the stability of least squares algorithms [1, 1] In [] it was shown that the lter coecients that maximize the output signal-to-noise ratio can be obtained from the eigenvector of X T X associated with its largest eigenvalue 5 :

10 8 Michael K Ng and William F Trench It can be shown that A n = X T X satises (1) with G n = 1 x m u 1 5 and H n = x m,n+ u n,x m v 1,x m,n+ v n 5 where u i = m,n+1 X l=1 x l x l+n,i and v j = mx l=j+1 x l x l,j : Therefore each iteration of Algorithm requires O(n ) operations 5 Numerical results We tried Algorithm on Toeplitz{block matrices (with s =)asmentioned in x and on matrices of the form T n T n where T n is an arbitrary real Toeplitz matrix The elements of these matrices are randomly generated with a uniform distribution in [ 1] All computations were done with Matlab in double precision Let 1 n be the eigenvalues of a Toeplitz-like matrix A n, and suppose we wish to nd i, where i is a specied integer in [1:::n] We assume that i is not an eigenvalue of any of the principal submatrices A 1 :::A n We rst nd an interval ( ) containing i but not any other eigenvalues of A n,orany eigenvalues of A n On such aninterval q n is continuous In [15] itwas shown that and satisfy this requirement if and only if Neg n () =i, 1 Neg n () =i q n () > and q n () < and a strategy was given for obtaining ( ) by means of bisection After ( ) is determined, we use the Matlab M-le \fzero" to nd i as a root of the function q n () (This root{nding algorithm was originated by T Dekker and further improved by R Brent see Matlab on-line documentation) We stop the iteration for i when the dierence between successive iterates k and k obtained by the root nder satises the inequality j k, k j 1 1 maxfj k j 1g: To check the accuracy of the individual eigenvalues and associated eigenvectors of the randomly generated Toeplitz{like matrices, we computed the residual norms i = ka ny n ( ~ i ), ~ i y n ( ~ i )k ky n ( ~ i )k where ~ i is the approximate ith eigenvalue and y n ( ~ i ) (as dened in () with = ~ i ) is an approximate i {eigenvector Tables 1 and show the distribution of f i g for 5 randomly generated matrices of order 1, 5 of order 5, and 5 of order 1, for two types of Toeplitz{like matrices Table lists the average number of iterations per eigenvalue for two types of Toeplitz{like matrices

11 EIGENVALUE PROBLEM FOR HERMITIAN TOEPLITZ{LIKE MATRICES 9 For each randomly generated Toeplitz{like matrix of order n we formed the diagonal matrix D n consisting of the computed eigenvalues and the matrix n whose columns are the corresponding computed eigenvectors For each matrix we computed the reconstruction and orthogonality errors = ka n, n D n T n k F ka n k F and = ki n, n T n k F p n where kk F is the Frobenius norm The results are shown in Tables and 5 We also tried Algorithm on matrices of the form X T X where X is as in (1), with m = 1 and n = 18 We considered 5 cases with fx 1 :::x 1 g generated by the second-order autoregressive (AR) process x k, 1:x k +:5x k, = k and 5 cases with fx 1 :::x 1 g generated by the second-order moving-average (MA) process x k = k +:5 k +:5 k, : In all instances f k g is a Gaussian process with mean zero and variance one, and E( j k )= jk Tables and show the distribution of the residual norm i and the relative error between the eigenvalues computed by Algorithm and those computed by the QR method, respectively Table 8 shows the values of and for these two input processes The average numbers of iterations per eigenvalue for the AR and MA processes were 1 and 15 respectively Table 1 Distribution of errors f i g for 5 matrices A n = T n Tn, where Tn are randomly generated nonsymmetric n n Toeplitz matrices Number of errors Interval n = 1 n = 5 n = 1 [1, 1 ) [1, 1, ) 1 [1, 1, ) 1 [1,5 1, ) 1 1 [1, 1,5 ) 5 1 [1, 1, ) [1,8 1, ) [1,9 1,8 ) [1 1,9 ) [1 1 1 ) [1 1 1 ) 5 [1 1 )

12 1 Michael K Ng and William F Trench Table Distribution of errors f i g for 5 randomly generated Toeplitz-block matrices with s =and n =m Number of errors Interval n = 1 n = 5 n = 1 [1, 1 ) [1, 1, ) [1, 1, ) 1 [1,5 1, ) 1 15 [1, 1,5 ) [1, 1, ) 91 9 [1,8 1, ) [1,9 1,8 ) [1 1,9 ) [1 1 1 ) [1 1 1 ) [1 1 ) 1 Table Average number of iterations per eigenvalue for computations summarized in Tables 1 and Number of iterations Type n = 1 n = 5 n = 1 Tn T n where T n are nonsymmetric Toeplitz matrices Toeplitz{block matrices Table Reconstruction and orthogonality errors for 5 matrices A n = T ntn where Tn are randomly generated nonsymmetric Toeplitz matrices n = 1 n = 5 n = 1 Interval [1,5 1, ) [1, 1,5 ) 1 1 [1, 1, ) 11 1 [1,8 1, ) [1,9 1,8 ) [1 1,9 ) 8 [1 1 1 ) Table 5 Reconstruction and orthogonality errors for 5 randomly generated Toeplitz-block matrices with s =and n =m n = 1 n = 5 n = 1 Interval [1, 1, ) 1 8 [1,8 1, ) [1,9 1,8 ) [1 1,9 ) [1 1 1 ) 5 [1 1 1 ) 1

13 EIGENVALUE PROBLEM FOR HERMITIAN TOEPLITZ{LIKE MATRICES 11 Table Distribution of errors f i g for 5 matrices X T X with m = 1 and n = 18 Number of errors Interval AR Process MA Process [1, 1, ) 1 [1,8 1, ) 8 1 [1,9 1,8 ) [1 1,9 ) [1 1 1 ) Table Distribution of the relative error between the eigenvalues computed by Algorithm : method and those computed by QR method for 5 matrices X T X with m = 1 and n = 18 Number of errors Interval AR Process MA Process [1, 1, ) [1,8 1, ) 1 1 [1,9 1,8 ) [1 1,9 ) 1 [1 1 1 ) 5 5 Table 8 Reconstruction and orthogonality errors for 5 matrices for X T X with m = 1 and n = 18 AR Process MA Process Interval [1,8 1, ) [1,9 1,8 ) [1 1,9 ) 5 8 [1 1 1 ) 1 1

14 1 Michael K Ng and William F Trench Summary The experimental results reported here show that Algorithm is an ecient and eective method for computing individual eigenvalues of Hermitian Toeplitz-like matrices For an n n Toeplitz{like matrix, the computational cost of each eigenvalue and an associated eigenvector is O(n ) operations The method is more ecient than general purpose methods such as the QR algorithm for obtaining a small number (compared to n) of eigenvalues (See [15]) Since the computation of each eigenvalue is independent of the computation of all others, the method is highly parallelizable Moreover, if q 1 (),, q n () are computed with a parallel processing machine utilizing as many processors as necessary to exploit the full parallelism in the algorithm, the multiplications as well as additions required to compute in (), (8) and (9) can be carried out simultaneously The inner products in (), (8) and (9) can also be computed simultaneously by employing parallel processors in O(log n) time units Therefore, the computations of fq 1 (),,q n ()g when performed by O(n) parallel processors, can be accomplished in O(n log n) time units Hence the computations of each eigenvalue, when performed by O(n) processors, can be accomplished in O(n log n) time REFERENCES [1] E T Browne, On the separation property of the roots of the secular equation, Amer J Math, 5 (19), pp 81{85 [] R Chan, J Nagy, and R Plemmons, Circulant Preconditioned Toeplitz Least Squares Iterations, SIAM J Matrix Anal Appl, 15 (199), pp 8{9 [] R Chan, J Nagy, and R Plemmons, Displacement Preconditioner for Toeplitz Least Squares Iterations, Elec Trans Numer Anal, (199), pp {5 [] R Chan and M Ng, Conjugate Gradient Methods for Solving Toeplitz Systems, SIAM Review, 8 (199), pp {8 [5] G Cybenko and C Van Loan, Computing the Minimum Eigenvalue of a Symmetric Positive Denite Toeplitz Matrix, SIAM J Sci Stat Comput, (198), pp 1{11 [] I Gohberg, T Kailath and V Olshevsky, Fast Gaussian Elimination with Partial Pivoting for Matrices with Displacement Structure, Math Comp, (1995), pp 155{15 [] S Haykin, Adaptive Filter Theory, nd ed, Prentice-Hall, Englewood Clis, NJ, 1991 [8] G Heinig and K Rost, Algebraic Methods for Toeplitz-like Matrices and Operators, Operator Theory: Advances and Applications V1, Birkhauser Verlag, Stuttgart, 198 [9] Y H Hu and S Y Kung, Toeplitz Eigensystem Solver, IEEE Trans Acoustics, Speech, Sig Proc, (1985), pp 1{11 [1] T Kailath, S Kung and M Morf, Displacement Ranks of Matrices and Linear Equations, J Math Anal Appl, 8 (199), pp 95{ [11] T Kailath and A Sayed, Displacement Structure: Theory and Applications, SIAM Review, (1995), pp 9{8 [1] N Levinson, The Wiener RMS (Root Mean Square) Error Criterion in Filter Design and Prediction, J Math and Phys, 5 (19), pp 1{8 [1] W Ferng, G Golub and R Plemmons, Adaptive Lanczos Methods for Recursive Condition Estimation, Numer Algo, 1 (1991), pp 1{19 [1] D Pierce and R Plemmons, Tracking the Condition Number for RLS in Signal Processing, Math Control Signals Systems, 5 (199), pp {9 [15] W F Trench, Numerical Solution of the Eigenvalue Problem for Hermitian Toeplitz Matrices, SIAM J Matrix Th Appl, 9 (1988), pp 91{ [1] W F Trench, Numerical Solution of the Eigenvalue Problem for Eciently Structured Hermitian Matrices, Linear Algebra Appl, 15{15 (1991), pp 15{ [1] J H Wilkinson, The Algebraic Eigenvalue Problem, Clarendon Press, 195