ETNA Kent State University

Transcription

1 Electroic Trasactios o Numerical Aalysis. Volume 2, pp , December 994. Copyright 994,. ISSN ETNA FAST ITERATIVE ETHODS FOR SOLVING TOEPLITZ-PLUS-HANKEL LEAST SQUARES PROBLES ICHAEL K. NG Abstract. I this paper, we cosider the impulse resposes of the liear-phase filter whose characteristics are determied o the basis of a observed time series, ot o a prior specificatio. The impulse resposes ca be foud by solvig a least squares problem mi d (X + X 2 )w 2 by the fast Fourier trasform (FFT) based precoditioed cojugate gradiet method, for ( +2 )- by- real Toeplitz-plus-Hakel data matrices X + X 2 with full colum rak. The FFT based precoditioers are derived from the spectral properties of the give iput stochastic process, ad their eigevalues are costructed by the Blackma-Tukey spectral estimator with Bartlett widow which is commoly used i sigal processig. Whe the stochastic process is statioary ad whe its spectral desity fuctio is positive ad differetiable, we prove that with probability, the spectra of the precoditioed ormal equatios matrices are clustered aroud, provided that large data samples are take. Hece if the smallest sigular value of X + X 2 is of order O( α ), α>0, the the method coverges i at most O((2α+) log +) steps. Sice the cost of formig the ormal equatios ad the FFT based precoditioer is O( log) operatios ad each iteratio requires O( log) operatios, the total complexity of our algorithm is of order O( log +(2α +)log 2 + log ) operatios. Fially, umerical results are reported to illustrate the effectiveess of our FFT based precoditioed iteratios. Key words. least squares estimatios, liear-phase filter, Toeplitz-plus-Hakel matrix, circulat matrix, precoditioed cojugate gradiet method, fast Fourier trasform. AS subject classificatios. 65F0, 65F5, 43E0.. Itroductio. The cojugate gradiet (CG) method is a iterative method for solvig symmetric positive defiite systems Aw = d; see for istace Golub ad va Loa [4, pp ]. Whe A is a rectagular matrix with full colum rak, oe ca still use the method to fid the solutio to the least squares problem (.) mi d Aw 2 where 2 deotes the usual Euclidea orm. This ca be doe by applyig the method to the ormal equatios (.2) A T Aw = A T d. The covergece rate of the method depeds o the eigevalues of the ormal equatios matrix A T A; see Axelsso ad Barker [, pp ]. If the eigevalues of A T A cluster aroud a fixed poit, covergece will be rapid. Thus, to make the algorithm a useful iterative method, oe usually precoditios the system. That meas, istead of solvig the origial system (.2), we solve the precoditioed system P A T Aw = P A T d with precoditioer P. I this paper, we apply the precoditioed cojugate gradiet (PCG) method to solve structured least squares problems arisig from sigal processig applicatios, where the data matrix A is a rectagular Toeplitz-plus-Hakel matrix with full colum rak. A matrix T =(t jk )issaidtobetoeplitzift jk = t j k, i.e., T is costat alog its diagoals. A matrix H =(h jk ) is said to be Hakel if h jk = h j+k. Received arch 25, 994. Accepted for publicatio November 8, 994. Commuicated by R. J. Plemmos Departmet of athematics, The Chiese Uiversity of Hog Kog, Shati, Hog Kog. 54

2 ichael K. Ng 55.. Liear-phase Filterig. Least squares estimatios have bee used extesively i a wide variety of applicatios i sigal processig, as for istace spectrum aalysis [, 9], system idetificatios [20], equalizatios [3] ad speech processig [5, p. 49]. I these applicatios, oe usually uses filters to estimate the trasmitted sigal from a sequece of received sigal samples or to model a ukow system. Oe importat class of filters commoly used i sigal processig is the class of fiite impulse respose (FIR) liear-phase filters. Such filters are especially importat for applicatios where frequecy dispersio, due to oliear phase, is harmful, as for example i speech processig. I this paper, we develop a impulse respose vector w of the liear-phase filter whose characteristics are determied o the basis of a observed time series, ad ot o a priori specificatio. It was show i [6, 2] that give real data samples {x(),x,...,x()} ad a desired respose vector d, the impulse resposes ca be foud by solvig the Toeplitz-plus-Hakel least squares problem (.3) mi d (X + X 2 )w 2. Here X is a ( +2 )-by- rectagular Toeplitz matrix with its first row ad colum give by [x(), 0,...,0] ad [x(),x,...,x(), 0,...,0] T. respectively. oreover, X 2 is a ( +2 )-by- rectagular Hakel matrix with its last colum give by [0,...,0,x(),x,...,x(), 0,...,0] T ad a zero vector as its first row..2. Outlie. The use of cojugate gradiet methods with circulat precoditioers for solvig -by- Toeplitz systems T z = v has bee studied extesively i recet years; see [6], [8], [9] ad [0]. Sice circulat matrix ca always be diagoalized by the discrete Fourier matrix, a -by- liear system with circulat coefficiet matrix ca be solved i O( log ) operatios, usig fast Fourier trasform (FFT). Also, matrix-vector multiplicatios T u ca be computed by usig FFT i O( log ) operatios, by first decomposig T ito a sum of circulat ad skew-circulat matrices; see Cha ad Ng [6]. It follows that the umber of operatios per iteratio of the precoditioed cojugate method is of order O( log ) operatios. I the practical applicatios, oe always assumes that the data matrix X +X 2 is of full colum rak. Therefore, the ormal equatios matrix (X + X 2 ) T (X + X 2 )is o-sigular ad positive defiite, ad the solutio w of (.3) is obtaied by solvig the ormal equatios (.4) (X + X 2 ) T (X + X 2 )w =(X + X 2 ) T d. Note that the ormal equatios matrix (X + X 2 ) T (X + X 2 )isa-by- Toeplitzplus-Hakel matrix T + H. By trasformig the Hakel matrix H to a Toeplitz matrix usig the reversal matrix J, the Hakel matrix-vector products H u ca be computed by usig FFT i O( log ) operatios. I this paper, we apply the precoditioed cojugate gradiet algorithm with circulat (FFT based) precoditioers to solve the ormal equatios (T + H )w = (X + X 2 ) T d. The mai result of the paper is that, uder some practical sigal

3 56 Toeplitz-plus-Hakel least squares problems processig assumptios, the spectrum of the Hakel matrix H is clustered aroud zero with probability. The cotributio of the term H is ot sigificat as far as the cojugate gradiet method is cocered, ad we therefore do ot approximate it by a circulat matrix. Thus, the precoditioer c(t ) is just defied to be the miimizer of Q T F over all -by- circulat matrices Q. Here F deotes the Frobeius orm. As T is a Toeplitz matrix, the circulat precoditioer c(t ) ca be foud i O( log ) operatios. We also show that the eigevalues of c(t )ca be derivedfrom the Blackma-Tukey spectral estimator with the Bartlett widow that is a commoly used o-parametric spectral estimatio method i sigal processig. As for the covergece rate of the method, we prove that if the stochastic process {x(i)} is statioary ad its uderlyig spectral desity fuctio is (l +)- times differetiable fuctio for l>0, the the spectra of the precoditioed matrices c(t ) (T + H ) are clustered with probability. If the smallest sigular value of X + X 2 is of order O( α ) with α>0, the method coverges i at most O((2α +)log + ) steps with probability. Sice the data matrices X ad X 2 are Toeplitz ad Hakel matrices respectively, the ormal equatios ad the circulat precoditioer ca be formed i O( log ) operatios; see Ng ad Cha [23]. Oce they are formed, the cost per iteratio of the precoditioed cojugate gradiet method is of order O( log ) operatios, as oly Toeplitz, Hakel ad circulat matrix-vectors multiplicatios are required i each iteratio. Therefore, the total work of obtaiig the impulse resposes to a give accuracy is of order O( log +(2α +)log 2 + log ). The outlie of the paper is as follows. I Sectio 2, we study some properties of the ormal equatios matrices ad itroduce our FFT based precoditioers. I Sectio 3, we aalyze the covergece rate of the method probabilistically. I Sectio 4, umerical experimets are discribed which illustrate the effectiveess of the method. Some cocludig remarks are give i Sectio FFT based Precoditioers. The least squares solutios to (.3) ca be obtaied by solvig the scaled (ormalized versio of) ormal equatios (2.) 2 (XT X + X2 T X 2 + X2 T X + X T X 2 )w = 2 (X + X 2 ) T x. We ote that X ad X 2 have special structures. Each row of X is a right-shifted versio of the previous row ad each row of X 2 is a left-shifted versio of the previous row. By utilizig these special rectagular Toeplitz ad Hakel structures, the matrices 2 (XT X + X2 T X 2)ad 2 (XT 2 X + X T X 2) ca be writte as 2 (XT X + X2 T (2.2) X 2 )=T ad 2 (XT 2 X + X T X 2 )=H, respectively. Here T is a -by- symmetric Toeplitz matrix ad H is a -by- symmetric Hakel matrix. The first colum of T is give by [γ 0,γ,...,γ ] T, ad the first row ad the last colum of H are give by [γ 2,γ 2 2,...,γ ] ad [γ,γ,...,γ ] T, respectively, where γ k = k j= x(j)x(j + k), k =0,,...,2.

4 ichael K. Ng 57 I the statistics literature, if the iput stochastic process is statioary, the parameters γ k are called estimators of the autocorrelatio of the statioary process. The parameters γ k have a smaller mea square error tha other estimators; see for istace Priestley [24, p. 322]. Our precoditioer is take to be the circulat approximatio of the Toeplitz part T of the ormal equatios matrix. We remark that our precoditioer is differet from that recetly proposed by Ku ad Kuo [8] for Toepltiz-plus-Hakel systems. They basically take the circulat approximatios of Toepltiz matrix ad Hakel matrix ad the combie them together to form a precoditioer. We ote that uder the assumptios i [8], the spectrum of the Hakel matrix is ot clustered aroud zero. The motivatio behid our precoditioer is that the Toeplitz matrix T is the sample autocorrelatio matrix which ituitively should be a good estimatio to the autocorrelatio matrix of the discrete-time statioary process, provided that a sufficietly large umber of data samples are take. oreover, we prove i 3 that uder practical sigal processig assumptios, the spectrum of the Hakel matrix H is clustered aroud zero. Hece it suffices to approximate T by circulat precoditioer. I this paper, we oly focus o a optimal circulat precoditioer c(t )fort which is defied to be the miimizer of Q T F over all -by- circulat matrices Q ; see T. Cha [0]. The (j, k) etry of c(t ) is give by the diagoals c j k where (2.3) { ( k)γk + kγ k c k =, 0 k, c +k, 0 < k <. As T is a Toeplitz matrix, the circulat precoditioer c(t ) is foud i O() operatios. A iterestig spectral property of c(t )isthatift is symmetric positive defiite, the correspodig optimal circulat matrix c(t ) is also symmetric positive defiite. I fact, we have that (2.4) λ mi (T ) λ mi (c(t )) λ max (c(t )) λ max (T ), where λ mi ad λ max deote the miimum ad maximum eigevalues respectively; see Tyrtyshikov [26]. I additio, the precoditioer is closely related to the Blackma-Tukey spectral estimator with the Bartlett widow that is oe of the popular method for oparametric spectral aalysis i sigal processig; see [2]. The Bartlett spectral estimator ca be expressed as s(ω) = W (k)γ k e iωk, k= ω [0, 2π], where { W (k) = k, k, 0, k >; see [7, p. 80]. O the other had, as c(t ) is a circulat matrix, it ca be diagoalized by the discrete Fourier matrix F with etries [F ] j,k = e 2πijk/,wehavethat c(t )=F Λ F,

5 58 Toeplitz-plus-Hakel least squares problems where Λ is a diagoal matrix whose diagoal etries are the eigevalues of c(t ). Usig the relatioship betwee the first colum of c(t ) ad its eigevalues, the eigevalues λ j (c(t )) of c(t ) ca be expressed as [ k λ j (c(t )) = γ 0 + γ k + k ] (2.5) γ k ξj k, 0 j<, k= where ξ j = e 2πij/ ; see also Cha ad Yeug [9]. After some rearragemet of the terms i (2.5), we ote that the eigevalues of c(t ) are equal to the values of s(ω) sampled at the poits {2πj/} j=0 o [0, 2π]. 3. Spectra of Precoditioed Normal Equatios atrices. As we deal with data samples from stochastic processes, the covergece rate will be cosidered i a probabilistic way which is differet from the determiistic case discussed i [, pp ]. We first make the followig practical sigal processig assumptios (A) o the iput discrete-time real-valued process {x(i)}: (A) The process is statioary with o-zero costat mea µ; (A2) The uderlyig spectral desity fuctio of the process is a (l + )-times differetiable fuctio for l>0; (A3) The spectral desity fuctio of the process is positive; k (A4) The variaces of j= x(j) ad k j= [x(j) µ][x(j + k) µ] are bouded by Var k (3.) x(j) β, k =0,, 2,...,, j= ad Var k (3.2) [x(j) µ][x(j + k) µ] β 2, k =0,, 2,...,, j= where β ad β 2 are positive costats depedig o the iput stochastic process. The remarks o the assumptios ca be foud i [23]. The followig lemma, which gives the spectrum of the covariace matrix, R appeared i Hayki [5, p.39]. Lemma 3.. Let the stochastic process {x(i)} be statioary with zero mea ad let its spectral desity fuctio be f(θ) with miimum ad maximum values f mi ad f max, respectively. The the spectrum σ(r ) of R satisfies (3.3) σ(r ) [f mi,f max ],. I the followig, we express x(j)x(j + k) itermsofµ: x(j)x(j + k) =[x(j) µ][x(j + k) µ]+µ[x(j)+x(j + k)] µ 2. Thus, the matrices T ad H ca be writte as T = T () + µt µ 2 T (3) ad H = H () + µh µ 2 H (3), respectively. The (j, k)th etries of Toeplitz matrices T (), T ad T (3) are give by [T () ] j,k = j k (3.4) [x(p) µ][x(p + j k ) µ], 0 j, k <, p=

6 ichael K. Ng 59 (3.5) [T ] j,k = j k [x(p)+x(p + j k )], 0 j, k <, p= ad (3.6) [T (3) ] j,k = ( j k ), 0 j, k <, respectively. The (j, k)th etries of Hakel matrices H (), H ad H (3) are give by (3.7) [H () ] j,k = 2++j+k [x(p) µ][x(p +2 j k) µ], 0 j, k <, p= (3.8) [H ] j,k = 2++j+k [x(p)+x(p +2 j k)], 0 j, k <, p= ad (3.9) [H (3) ] j,k = ( 2 ++j + k), 0 j, k <, respectively. By the liearity of the optimal circulat approximatio, c(t )isdecomposed ito three parts: (3.0) c(t )=c(t () )+µc(t ) µ2 c(t (3) ). I the followig discussios, we let E(Z) be the expected value of a radom matrix Z, so that the etries of E(Z) are the expected value of the elemets of Z, i.e., k=τ j= [E(Z)] j,k = E([Z] j,k ), 0 j, k <. The followig two lemmas will be useful later i the aalysis of the covergece rate of the method. Lemma 3.2. ( Ng ad Cha [23, Theorem ] ) Let the stochastic process {x(i)} satisfy assumptios (A), (A2) ad (A4). The for ay give ɛ>0 ad 0 <η<, there exist positive itegers K ad N such that for >N, E(T () ) R 2 ɛ, ad Pr { at most K eigevalues of T () c(t () ) have absolute value greater tha ɛ } > η, provided that =Ω( 3+ν ) with ν>0, i.e., umber of data samples take is at least as large as 3+ν. We remark that i Lemma 3.2, the parameter ν is theoretically used to let the probability of the evet ted to. Lemma 3.3. Let the stochastic process {x(i)} satisfy assumptio (A4). The, for ay give ɛ>0, we have that τ 2 k Pr [x(j)+x(j + k)] E k [x(j)+x(j + k)] ɛ j= 8 τ 2 τ + 3 β ɛ 2,

7 60 Toeplitz-plus-Hakel least squares problems ad that τ 2 k Pr [x(j) µ][x(j + k) µ] E k [x(j) µ][x(j + k) µ] ɛ k=τ j= j= τ 2 τ + 3 β 2 ɛ 2, for ay itegers τ ad τ 2. Proof. By usig a lemma i Fuller [2, p.82] ad Chebyshev s iequality [2, p.85], we have τ 2 k Pr [x(j)+x(j + k)] E k [x(j)+x(j + k)] k=τ j= j= ɛ τ 2 k [x(j)+x(j + k)] E k [x(j)+x(j + k)] ɛ k=τ Pr τ 2 k=τ j= ( 4 τ 2 τ + [Var 2 j= k j= x(j) ) +Var ɛ 2 ( k j= x(j + k) )] τ 2 τ + 8 τ 2 τ + 3 β ɛ 2. The other part ca be derived similarly; it is therefore omited. Usig Lemma 3.3, we prove that the l 2 orm of the differece betwee the radom matrices ad their expected values is sufficietly small with probability. Corollary 3.4. Let the stochastic process {x(i)} satisfy assumptios (A), (A2) ad (A4). The, for ay give ɛ>0 ad 0 <η<, we have that (3.) Pr { T () () E(T )+µ[t E(T )] 2 ɛ } > η ad that (3.2) { Pr H () E(H () )+µ[h E(H } )] 2 ɛ > η, provided that =Ω( 3+ν ) with ν>0. Proof. Weotethat T () E(T () )+µ[t E(T )] 2 T () E(T () ) 2 + µ T E(T ) 2 T () () E(T ) + µ T E(T ). It ca be show that both T () E(T () ) ad T E(T ) are bouded by 2 times the l orm of their correspodig first colum vectors; see (3.4) ad (3.5). The the result follows by settig τ =0adτ 2 = i Lemma 3.3. Usig similar argumets, we establish the same boud for H () E(H () )+H () E(H () ) 2.I fact, H () E(H () ) ad H E(H ) are bouded by 2 times the l orm of their correspodig last colum vectors. Hece, (3.2) follows.

8 ichael K. Ng 6 Next we prove the mai result of the paper about the clusterig property of the matrices H. Theorem 3.5. Let the stochastic process {x(i)} satisfy assumptios (A), (A2) ad (A4). The for ay give ɛ>0 ad 0 <η<, there exist positive itegers K ad N such that for >N,Pr{ at most K eigevalues of H have absolute value greater tha ɛ } > η, provided that =Ω( 3+ν ) with ν>0. Proof. WewriteH as follows: H = H () E(H () )+µ[h E(H )] µ 2 H (3) + µe(h L + µ 2 L + E(H () where L is a -by- matrix with all etries beig. By (3.8) ad (A), we obtai µe(h ) µ2 H (3) = µ2 H (3). By usig (3.2) i Corollary 3.4 ad H (3) { Pr H () E(H () )+µ[h E(H )] µ 2 H (3) L 2 ( ),wehavethat } + µe(h ) µ 2 L 2 ɛ > η, provided that = Ω( 3+ν ) with ν > 0. We remark that the rak (L ) =. Therefore, it suffices to prove that the spectrum of E(H () ) is clustered aroud zero determiistically. By (3.7), the etries of E(H () ) are give by [E(H () )] j,k = ( 2 ++j + k)r 2 j k, 0 j, k <, where r k is the k-lag autocovariace of the statioary process. By (A2), the autocovariaces of the statioary process are absolutely summable. Hece, for ay give ɛ>0, there exists a N>0 such that (3.3) j=n+ r j <ɛ. Let U be the -by- matrix obtaied from E(H () ) by replacig the ( N)-by- ( N) leadig pricipal submatrix of E(H () ) by the zero matrix. The, rak (U ) 2N. LetV E(H () ) U. The leadig ( N)-by-( N) blockofv is the leadig ( N)-by-( N) pricipal submatrix of E(H () ). Hece, this block is a Hakel matrix, ad usig (3.3) ad r k β 3 k l+, where β 3 is a positive costat, the l orm of V is attaied at the ( N )th colum. As V is a symmetric matrix, the result follows by otig that V 2 V ɛ. Uder the assumptios, the smallest eigevalues of T ad c(t ) are uiformly bouded away from zero with probability. Therefore, c(t ) is uiformly ivertible. As we cosider the process with o-zero mea i geeral, the theorem below exteds the result of Theorem 2 i Ng ad Cha [23].

9 62 Toeplitz-plus-Hakel least squares problems Theorem 3.6. Let the stochastic process {x(i)} satisfy assumptios (A) ad (A4). The for ay give ɛ>0 ad 0 <η<, there exist a positive iteger N such that for >N, Pr {λ mi (T ) f mi ɛ} > η 2 ad Pr { λ max (T ) µ 2 + f max + ɛ } > η 2, provided that =Ω( 3+ν ) with ν>0. I particular, we have that Pr {λ mi (c(t )) f mi ɛ} > η 2 ad Pr { λ max (c(t )) µ 2 + f max + ɛ } > η 2. Proof. Wewrite T = T () () E(T )+µ[t E(T )] µ2 T (3) + µe(t ) µ2 L + E(T () ) R + R + µ 2 L. By (3.5), (3.6) ad (A), we obtai µe(t ) µ 2 T (3) = µ 2 T (3) ad T (3) ( ) L 2. Usig Lemma 3. ad the fact that R ad L are symmetric ad the eigevalues of µ 2 L are 0 ad µ 2, it follows by Corollary i [4, p.269] that the smallest ad largest eigevalues of R +µ 2 L are bouded below by f mi ad above by f max +µ 2, respectively. The, the result follows by usig Lemma 3.2, (3.) i Corollary 3.4 ad some simple probability argumets, provided that =Ω( 3+ν ) with ν>0. Usig (2.4), we immediately have the result for the smallest ad largest eigevalues of c(t ). Theorem 3.7. Let the stochastic process {x(i)} satisfy assumptios (A), (A2) ad (A4). The for ay give ɛ>0 ad 0 <η<, there exist positive itegers K ad N such that for >N,Pr{ at most K eigevalues of T + H c(t ) have absolute value greater tha ɛ } > η 2, provided that =Ω(3+ν ) with ν>0. Proof. By (3.0), T c(t ) = T () c(t () )+µ[t E(T )+E(T ) E(c(T )) + E(c(T )) c(t )] µ2 [T (3) (3) c(t )]. We ote from (3.5), (3.6) ad (A) that µe(t () 2 ) µ 2 T (3) = µ 2 T (3) ad µe(c(t () 2 )) + µ 2 c(t (3) )= µ 2 c(t (3) ). Thus, (3.4) becomes T c(t ) = T () () c(t )+µ[t E(T µ 2 T (3) L + L µ 2 c(t (3) ). )+c(t ) E(c(T ))] + Usig (2.4), the commutative property of circulat approximatio ad expectatio operator ad the circulat structure of L,weobtai (3.4) c(t ) E(c(T )) 2 T E(T ) 2

10 ichael K. Ng 63 ad (3.5) c(t (3) ) L 2 T (3) ( ) L 2. I view of Lemmas 3.2 ad 3.5, (3.) i Corollary 3.4, (3.4) ad (3.5), the theorem follows. By combiig Theorems (3.6) ad (3.7), the mai theorem cocerig the spectra of the precoditioed matrices is proved. Theorem 3.8. Let the stochastic process {x(i)} satisfy assumptio (A). The for ay give ɛ>0 ad 0 <η<, there exist positive itegers K ad N such that for >N,Pr{ at most K eigevalues of c(t ) (T + H ) have absolute value larger tha ɛ } > η, provided that =Ω( 3+ν ) with ν>0. As for the covergece rate of the precoditioed cojugate gradiet method for our circulat precoditioed Toeplitz-plus-Hakel matrix c(t ) (T + H ), the method coverges i at most O((2α + )log + ) steps whe the smallest sigular value of the data matrix X + X 2 is of order O( α ). We begi by otig the followig error estimate of the cojugate gradiet method; see [5]. Lemma 3.9. Let G be a -by- positive defiite matrix ad z be the solutio to G z = v. Letz j be the jth iterat of the ordiary cojugate gradiet method applied to the equatio G z = v. If the eigevalues {λ k } of G are such that 0 <λ... λ p b λ p+... λ q b 2 λ q+... λ, the (3.6) ( ) { j p q p z z j G b 2 max z z 0 G b + λ [b,b 2] k= ( ) } λ λk. λ k Here b (b 2 /b ) 2 ad v G v G v. For the precoditioed system (3.7) c(t ) (T + H )w =(X + X 2 ) T d, the iteratio matrix G is give by G = c(t ) /2 (T + H )c(t ) /2. Theorem 3.8 implies that we ca choose b = ɛ ad b 2 =+ɛ, with probability. The, p ad q are costats that deped oly o ɛ but ot o. Bychoosigɛ<, we have that b b + = ɛ 2 <ɛ. ɛ I order to use (3.6), we eed a lower boud for λ k, k p. We ote that with probability G 2 = (T + H ) c(t ) 2 c(t ) 2 (T + H ) 2 ( + f max + ɛ) 2α. We the see that for all sufficietly large, G 2 β 4 2α+, for some costat β 4 that does ot deped o. Therefore, λ k mi λ l = l G β 4 (2α+), k. 2

11 64 Toeplitz-plus-Hakel least squares problems Thus, for k p ad λ [ ɛ, +ɛ], we have that Hece, (3.6) becomes 0 λ λ k λ k β 4 2α+. w w j G w w 0 G <β p 4 p(2α+) ɛ j p q. Give a arbitrary tolerace δ > 0, a upper boud for the umber of iteratios required to make is therefore give by w w j G w w 0 G <δ j 0 p + q p log β 4 +(2α +)p log log δ log ɛ = O(2α log +), with probability. Sice by usig FFTs, the Toeplitz, Hakel ad circulat matrix-vector products i the PCG method ca be doe i O( log ) operatios, the cost per iteratio of the cojugate gradiet method is of order O( log ). Thus, we coclude that the work of solvig (3.7) to a give accuracy δ is of order O((2α +)log 2 + log ) whe α>0. We remark that the order of complexity of our iterative method is less tha that of direct methods (see erchat ad Parks [22] ad Yagle [27]) which requires O( 2 ) operatios. 4. Numerical Experimets. I this sectio, the results of umerical experimets which test the covergece performace of the algorithm are described. All the computatios are doe by atlab o a Sparc workstatio. We used AR ad A processes give by x(t).4x(t ) + 0.5x(t 2) = v(t) ad x(t) =v(t)+0.75v(t ) v(t 2), respectively, to geerate the Toeplitz-plus-hakel matrices T + H. Here {v(t)} is a Gaussia process with zero mea ad variace as iput statioary process. I Figures ad 2, we depict the spectra of the ormal equatios matrix ad the precoditioed ormal equatios matrix i oe of the realizatios of the AR ad A processes, respectively, with = 28 ad = 024. We ote that the spectra of the precoditioed matrices ideed are clustered aroud. I the umerical tests, we uses the zero vector ad a radom vector as our iitial guess ad right had side vector. The stoppig criterio of the precoditioed cojugate gradiet method was e j 2 / e 0 2 < 0 7,wheree j is the residual vector after j iteratios. I the tables below, = / is the umber of blocks of data samples with size ad I deotes o precoditioer is used whereas C sigifies the optimal circulat precoditioer is used. Tables 2 show the average umber of iteratios (rouded to the earest iteger) over 00 rus of the algorithms whe AR ad A processes are used. We see that the precoditioed system coverges very fast ad the average umber of iteratios of precoditioed systems is much less tha that of o-precoditioed oe whe is large. As for the compariso of times i

12 ichael K. Ng 65 2 =28, = precoditioed matrix ormal equatios matrix Fig. 4.. Eigevalues for ormal equatios matrix ad precoditioed matrix whe AR process is used. (autocorrelatio widowig method) I C I C I C I C Table 4. Average umber of iteratios whe AR process is used. cojugate gradiet iteratios, Tables 3 ad 4 show the average umber of kilo-flops (couted by atlab) used for the cases of the AR ad A processes tested i Tables ad 2. We see from the tables that the umber of kilo-flops used for the precoditioed systems is sigificatly less tha that of o-precoditioed systems especially whe is large. I this paper, we employed the autocorrelatio widowig method to formulate the Toeplitz-plus-Hakel least squares problem. Other widowig methods ca be used, for istace, the covariace widowig method, the pre-widowed method ad the post-widowed method; see Hayki [5, p.373]. We remark that the other widowig methods lead to o-toeplitz-plus-hakel ormal equatios matrices. However, by

13 66 Toeplitz-plus-Hakel least squares problems I C I C I C I C Table 4.2 Average umber of iteratios whe A process is used. 2 =28, = precoditioed matrix ormal equatios matrix Fig Eigevalues for ormal equatios matrix ad precoditioed matrix whe A process is used. (autocorrelatio widowig method) I C I C I C I C Table 4.3 Average umber of kilo-flops (rouded to earest kilo-flops) whe AR process is used.

14 ichael K. Ng I C I C I C I C Table 4.4 Average umber of kilo-flops (rouded to earest kilo-flops) whe A process is used I C I C I C I C Table 4.5 Average umber of iteratios whe AR processes is used. exploitig the structure of the ormal equatios matrices, it ca still be writte as 2 (X + X 2 ) T (X + X 2 )=T + H S () S S(3) S(4), where S (i) are o-toeplitz ad o-hakel matrices. By cosiderig similar argumets as i Ng ad Cha [23], it ca be show that the l 2 orm of these matrices S i (i =, 2, 3, 4) are sufficietly small whe is sufficietly large. Therefore, our algorithm ca hadle the Toeplitz-plus-Hakel least squares problems with the use of differet widowig methods. To illustrate the performace of our precoditioer for these problems, we use the AR process to geerate the covariace widowig data matrices X ad X 2. I Figure 3, we depict the spectra of the ormal equatios matrix ad the precoditioed ormal equatios matrix i oe realizatio of the AR process where = 28 ad = 024. The figure shows clusterig of the eigevalues of the FFT based precoditioed matrices. Also Tables 5 ad 6 show the average umber of iteratios (rouded to the earest iteger) ad the correspodig average umber of kilo-flops required respectively over 00 rus of the algorithms whe the AR process is used. We see that both the average umber of iteratios ad the average kilo-flops used by the precoditioed systems are much less tha those of the o-precoditioed systems especially whe is large. 5. Cocludig Remarks. I this paper, we have proposed a ew FFT based precoditioed Toeplitz-plus-Hakel least squares iteratio. Our prelimiary umerical results show the effectiveess of our algorithm. As a summary, we list the followig remarks cocerig our algorithm: (i) I sigal processig applicatios, the liear-phase filters ca also be characterized by atisymmetric impulse resposes. We solve the Toeplitz-plus-Hakel

15 68 Toeplitz-plus-Hakel least squares problems 2 =28, = precoditioed matrix ormal equatios matrix Fig Eigevalues for ormal equatios matrix ad precoditioed matrix whe AR process is used. (covariace widowig method) I C I C I C I C Table 4.6 Average umber of kilo-flops (rouded to earest kilo-flops) whe AR process is used. least squares problem ad the ormal equatios become mi d (X X 2 )w 2, 2 (XT X + X2 T X 2 X2 T X X T X 2)w = 2 (X X 2 ) T x. The precoditioed cojugate gradiet algorithm ca also be applied to solve ormal equatios i this case.

16 ichael K. Ng 69 (ii) Recetly, other discrete trasform matrices W have bee used to costruct the optimal precoditioers to symmetric Toeplitz matrices. These trasform matrices iclude the sie trasform [7] ad the Hartley trasform [3]. These precoditioers are defied to be the miimizer of Q T F over all -by- matrices Q that ca be diagoalized by W. We ote that they are defied similarly to c(t ). I [7] ad [3], it was show that the these precoditioers perform very well whe solvig symmetric Toeplitz systems. Thus, we expect these precoditioers to be good alteratives to our FFT based oes for solvig Toeplitz-plus-Hakel least squares problems. (iii) I [22], it has bee show that a Toeplitz-plus-Hakel system of equatios ca be reformulated as a block-toeplitz system of equatios with 2 2 blocks, i.e. ( T H )( w H T J w ) = I this case, a block-circulat precoditioer ( c(t ) 0 0 c(t ) ( d J d ca be used to precoditio the block equatios. By Theorem 3.5, we ote that the block-circulat matrix is also a good precoditioer. However, the approach doubles the dimesio of the problem beig solved, ad hece it doubles the operatios per iteratio. (iv) Our algorithm preseted i this paper is of the fixed order ad the blockprocessig type, i.e. data samples are collected over a fiite time iterval; the estimates of the autocorrelatios are the computed ad a -by- Toeplitz-plus-Hakel system as i (2.) is formed ad solved by the precoditioed cojugate gradiet method. The complexity of solvig Toeplitz-plus- Hakel systems is O( log 2 ) operatios as compared to O( 2 ) operatios required by direct solvers. We ote that the basic tool of our fast iterative algorithm is the fast Fourier trasform (FFT). Sice the FFT algorithm is highly parallelizable ad has bee implemeted o multiprocessors efficietly (see for istace Swarztrauber [25]), our algorithm is expected to perform efficietly i a parallel eviromet. 6. Ackowledgemet. The author thaks the referees for umerous valuable commets ad helpful suggestios. ) ). REFERENCES [] O. Axelsso ad V. Barker, Fiite Elemet Solutio of Boudary Value Problems, Theory ad Computatio, Academic, New York, 984. [2]. S. Barlett, Periodogram Aalysis ad Cotiuous Spectra, Biometrika, 37 (950), pp. 6. [3] D. Bii ad P. Favat, O a atrix Algebra Related to the Discrete Hartley Trasform, SIA J. atrix Aal. Appl., 4 (993), pp [4] P. J. Brockwell, Time Series: Theory ad ethods, Spriger-Verlag, New York, 987. [5] R. Cha, J. Nagy ad R. Plemmos, Circulat Precoditioed Toeplitz Least Squares Iteratios, to appear SIA J. atrix Aal. Appl., 5 (994), pp [6] R. Cha ad. Ng, Toeplitz Precoditioers for Hermitia Toeplitz Systems, Liear Algebra Appl., 90 (993), pp

17 70 Toeplitz-plus-Hakel least squares problems [7] R. Cha,. Ng ad C. Wog, Sie Trasform Based Precoditioers For Symmetric Toeplitz Systems, to appear i Liear Algebra Appl., (994). [8] R. Cha ad G. Strag, Toeplitz Equatios by Cojugate Gradiets with Circulat Precoditioer, SIA J. Sci. Statist. Comput., 0 (989), pp [9] R. Cha ad. Yeug, Circulat Precoditioers Costructed From Kerels, SIAJ. Numer. Aal., 29 (992), pp [0] T. Cha, A Optimal Circulat Precoditioer for Toeplitz Systems, SIA J. Sci. Statist. Comput., 9 (988), pp [] B. Friedlader ad. orf, Least Squares Algorithms for Adaptive Liear-Phase Filterig, IEEE Tras. Acoust. Speech ad Sigal Processig, 30 (982), pp [2] W. Fuller, Itroductio to Statistical Time Series, Joh Wiley & Sos, Ic., New York, 976. [3] D. Godard, Chael Equalizatios Usig a Kalma Filter for Fast Data Trasmissio, IB J. Res. Develop., 8 (974), pp [4] G. Golub ad C. Va Loa, atrix Computatios, Secod ed., The Johs Hopkis Uiversity Press, Baltimore, D, 989. [5] S. Hayki, Adaptive Filter Theory, Secod ed., Pretice-Hall, Eglewood Cliffs, NJ, 99. [6] J. Hsue ad A. Yagle, Fast Algorithms for Close-to-Toeplitz-plus-Hakel Systems of Equatios ad Two-Sided Liear Predictio, IEEE Tras. Sigal Processig 4 (993), pp [7] S. Kay, Spectral Estimatio, Advaced Topics i Sigal Processig, Pretice-Hall, Eglewood Cliffs, NJ, 988, pp [8] T. Ku ad C. Kuo, Precoditioed Iterative ethods for Solvig Toeplitz-plus-Hakel Systems, SIA J. Numer. Aal. 30 (993), pp [9] S. arple, A New Autoregressive Spectrum Aalysis Algorithm, IEEE Tras. Acoust. Speech ad Sigal Processig, 28 (980), pp [20], Efficiet Least Squares FIR System Idetificatio, IEEE Tras. Acoust. Speech ad Sigal Processig, 29 (98), pp [2], Fast Algorithms for Liear Predictio ad System Idetificatio Filters with Liear Phase, IEEE Tras. Acoust. Speech ad Sigal Processig, V30 (982), pp [22] G. erchat ad T. Parks, Efficiet Solutio of a Toeplitz-plus-Hakel Coefficiet atrix System of Equatios, IEEE Tras. Acoust. Speech ad Sigal Processig, 30 (982), pp [23]. Ng ad R. Cha, Fast Iterative ethods for Least Squares Estimatios, Numer. Algorithms, 6 (994), pp [24]. Priestley, Spectral Aalysis ad Time Series, Academic Press, New York, 98. [25] P. Swarztrauber, ultiprocessors FFTs, Parallel Comput., 5 (987), pp [26] E. Tyrtyshikov, Optimal ad Super-optimal Circulat Precoditioers, SIA J. atrix Aal. Appl., 3 (992), pp [27] A. Yagle, New Aalogues of Split Algorithms for Arbitrary Toeplitz-plus-Hakel atrices, IEEE Tras. Sigal Processig, 39 (99), pp