Tridiagonal reduction redux

Transcription

1 Week 15: Moday, Nov 27 Wedesday, Nov 29 Tridiagoal reductio redux Cosider the problem of computig eigevalues of a symmetric matrix A that is large ad sparse Usig the grail code i LAPACK, we ca compute all the eigevalues of a -by- tridiagoal matrix i O() time; but the usual Householder-based algorithm to reduce A to tridiagoal form costs O( 3 ) Though it s possible to maitai sparsity i some cases, it is ot altogether straightforward So let s see if we ca try somethig differet What we wat is a orthogoal matrix Q such that AQ = QT, where α 1 β 1 β 1 α 2 β 2 T = β 2 α 3 β 1 β 1 Writig colum j of this matrix equatio gives us α Aq j = β j 1 q j 1 + α j q j + β j q j+1, where α j = q T j Aq j ad β j 1 = q T j 1Aq j If we rearrage this slightly, we have r j+1 = Aq j q j α j q j 1 β j 1 = Aq j (q j q T j )Aq j (q j 1 q T j 1)Aq j q j+1 = r j+1 /β j That is, q j+1 is exactly what we compute if we orthogoalize Aq j agaist the vectors q j ad q j 1 Note that Aq j is automatically orthogoal to q 1,, q j 2, sice q T k Aq j = T kj is zero for k < j 1 This process of icremetally producig a orthoormal basis by multiplyig the last vector by A ad orthogoalizig the result agaist what came before gives us the basic Laczos procedure If we start with q 1 = e 1, the Laczos procedure will give us the same T matrix 1 as the Householder 1 Well, almost the same T matrix Note that the off-diagoal elemets β j might be positive or egative i the Householder tridiagoalizatio, but they are always positive i the Laczos algorithm However, the two T matrices satisfy a similarity relatio with a diagoal matrix whose diagoal etries are ±1

2 algorithm, at least i exact arithmetic But while a step i the Householder algorithm costs O( 2 ), the cost of a Laczos step is a matrix-vector multiply (which may cost less tha O( 2 ) if A is sparse) followed by O() work i dot products ad gaxpy operatios Also, ote that the Laczos step requires oly a few vectors of itermediate storage I exact arithmetic, we could iterate util we foud β k = 0, which would correspod to havig a ivariat subspace spaed by {q 1,, q k } Except for very special choices of startig vectors, though, this would happe oly for k = Partial tridiagoalizatio ad Krylov subspaces What happes whe we ru oly a few steps of the Laczos iteratio? After m steps, we have computed m colums of Q ad m rows of T ; ad there is some hope that we might be able to use the leadig m-by-m subblock of T (ie the block Rayleigh quotiet with the first m colums of Q) i order to extract some useful iformatio about the spectrum of A The reaso why we might have such a hope is that we otice that the Krylov subspace spaed by q 1,, q m is also spaed by the first m iterates of a power method: K m (A, q 1 ) = spa{q 1, q 2,, q m } = spa{q 1, Aq 1,, A m 1 q 1 } As we have previously discussed, the power basis is ot a particularly useful basis for umerical computatios, but it is useful for thikig about Krylov subspaces I particular, because it cotais the mth vector produced by power iteratio startig from q 1, we expect that the largest Ritz value associated with K m (A, q 1 ) (ie the largest magitude eigevalue of the block Rayleigh quotiet (Q :,1:m ) T AQ :,1:m ) will be at least as large as the Rayleigh quotiet estimate produced by m steps of power iteratio Of course, the Krylov subspace K m (A, q 1 ) has a lot more iformatio tha just the vector from step m of a power iteratio For example, otice that if σ is ay shift, the K m (A σi, q 1 ) = K m (A, q 1 ); that is, Krylov subspaces are shift ivariat This is relevat because addig a shift to A ca chage which of the exterior eigevalues i the spectrum is domiat So if λ 1 λ 2 λ, the we expect to be able to extract good estimates to λ 1 ad λ from a few steps of Laczos iteratio uder the assumptio that power iteratio with certai shifted matrices coverges sufficietly fast

3 Laczos i iexact arithmetic The observat reader will have oticed somethig usettlig about the basic Laczos iteratio: at the heart of the Laczos iteratio is a Gram- Schmidt orthogoalizatio step, which we said i our discussio of QR factorizatios could become umerically ustable I particular, whe r k+1 = (A α k )q k + q k 1 β k 1 is small (ie whe β k is small relative to A ), cacellatio may cause the computed vector to cosisty mostly of roudoff, so that it is o loger orthogoal to the precedig vectors But a small value of β k correspods exactly to covergece to a good approximatio to a ivariat subspace! Thus, i floatig poit arithmetic, the basic Laczos iteratio teds to ru reasoably well util covergece, at which poit it effectively restarts with a radom startig vector which is made up primarily of roudoff error So we get covergece of the Ritz values of the computed T to the first few exterior eigevalues; ad the more ghost Ritz values coverge to the first few exterior eigevalues; ad we cotiue o i this fashio for as log as we are willig to let thigs ru This ot state of affairs is ot etirely satisfactory, but we ca fix it by re-orthogoalizatio: that is, we improve the orthogoality of the computed basis by explicitly orthogoalizig Aq k agaist vectors to which it should already be orthoormal i exact arithmetic Oe approach is to use complete orthogoalizatio: compute Aq j ad the orthogoalize agaist all previous vectors (eg by a sequece of Householder trasformatios) This is expesive i terms of storage (ad data movemet) ad arithmetic, but it does work A less expesive solutio, selective orthogoalizatio, ivolves orthogoalizig agaist coverged eigevector estimates (Ritz vectors) I the iterest of time, we will ot discuss reorthogoalizatio further If you are iterested, though or if you are iterested i ay umber of other aspects of Laczos iteratio that we skip over I recommed lookig at Parlett s book The Symmetric Eigevalue Problem Stewart s book Matrix Algorithms, Vol 2: Eigesystems also has a ice treatmet Why did umerical istability ot arise whe we discussed CG? It s there behid the scees, ad it explais why ruig CG for steps geerally does ot provide the exact aswer to a liear system But obody sesible uses CG as a direct method ayhow; it is used as a iterative solver, ad it turs out that the loss of orthogoality does ot matter that much to the CG iteratio cotiuig to make progress

4 Partial tridiagoalizatio ad residual bouds Suppose AQ = QT, where T ii = α i, T i,i+1 = β i If we take m steps of Laczos iteratio, we geerate Q :,1:m = [ q 1 q 2 q m ] as well as the first m coefficiets (α i ) m i=1 ad (β i ) m i=1 Let us deote Q m ad T m as the leadig m colums of Q ad the leadig m m submatrix of T respectively; the writig the first m colums of AQ ad QT gives us (1) AQ m = Q m T m + β m q m+1 e T m Here T m is a block Rayleigh quotiet T m = Q T maq m, ad the eigevalues of T m (the Ritz values) are used to approximate the eigevalues of A Now cosider the eigedecompositio T m = Y ΘY T where Y T Y = I ad Θ = diag(θ 1,, θ m ) The postmultiplyig (1) by Y gives AQ m Y = Q m Y Θ + β m q m+1 e T my The colums of Z = Q m Y = [ z 1 z m ] are approximate eigevalues correspodig to the approximate eigevalues θ 1, θ 2,, θ m For each colum, we have Az k z k θ k = β m q m+1 e T my k, which meas Az k z k θ k 2 = β m e T my k This is useful because, as we discussed before, i the symmetric case a small residual error implies a small distace to the closest eigevalue This is also useful because the residual error ca be computed with o further matrix operatios we eed oly to look at quatities that we would already compute i the process of obtaiig the tridiagoal coefficiets ad the correspodig Ritz values I particular, ote that we ca compute the residual for the Ritz pair (approximate eigepair) (z k, θ k ) without explicitly computig z k! The polyomial coectio Suppose λ 1 λ 2 λ are the eigevalues of A, with correspodig eigevectors w 1 through w I matrix form, the, A = W ΛW T

5 Now, suppose we ru m steps of Laczos iteratio (i exact arithmetic), ad let θ 1 be the largest eigevector of T m (the largest Ritz value) This is the same as sayig that θ 1 maximizes the Rayleigh quotiet over the Krylov subspace K m (A, q 1 ): θ 1 = max y R m y T T m y y T y (Q m y) T A(Q m y) = max y R m (Q m y) T (Q m y) = max z T Az z K m(a,q 1 ) z T z Now, ay vector z K m (A, q 1 ) ca be writte as z = c 0 q 1 + c 1 Aq c m 1 A m 1 q 1 = p(a)q 1, where p is a polyomial of degree at most m 1 Thus, θ 1 = d T p(λ)λp(λ)d max deg(p)<m d T p(λ) 2 d q1 T p(a)ap(a)q 1 max deg(p)<m q1 T p(a) 2 q 1 This is still somewhat awkward, so let us rewrite thigs i terms of the eigevector basis Defie d = Z T q 1 ; the i=1 θ 1 = d2 i p(λ i ) 2 λ i i=1 d2 i p(λ i) 2 = max deg(p)<m Let s give a ame to the fuctio i this maximizatio: φ(p) = i=1 d2 i p(λ i ) 2 λ i i=1 d2 i p(λ i) 2 = λ 1 d2 i p(λ i ) 2 (λ 1 λ i ) i=1 d2 i p(λ λ i) 2 1 (λ 1 λ ) d2 i p(λ i ) 2 d 2 1p(λ 1 ) 2 We kow that θ 1 = max deg(p)<m φ(p) λ 1 ; we would also like a lower boud o θ 1 If we ca show that this lower boud approaches λ 1 at some rate with icreasig m, the we kow θ 1 will coverge to λ 1 at least as fast Note that if we foud a polyomial that was zero at λ 2,, λ ad ozero at λ 1, the we would recover λ 1 exactly But such a polyomial usually has too high a degree What we would like i order to get a boud, the, is a degree m polyomial which is ot too big o [λ 2, λ ], but is relatively large at λ 2 Havig see it s appearace i our aalysis of Chebyshev iteratio ad CG, you will ot be surprised that a good cadidate is our old fried, the rescaled Chebyshev polyomial ˆT k (x) = T k ( x λ λ 2 λ )

6 Note that ˆT k (λ j ) 1 for j > 1 ad ˆT k (λ 1 ) grows rapidly with k Note that for ay p, we have φ(p) = λ 1 d2 i p(λ i ) 2 (λ 1 λ i ) i=1 d2 i p(λ i) 2 λ 1 (λ 1 λ ) d2 i p(λ i ) 2 d 2 1p(λ 1 ) 2 Now, ote that i=1 d2 i = d 2 2 = q = 1, that ˆT m 1 (λ i ) 2 1 for i = 2,, ad that ˆT m 1 (λ 1 ) = T c m 1 (1 + 2ρ 1 ) Thus, φ(ĉ m 1 ) λ 1 (λ 1 λ ) d2 ˆT i m 1 (λ i ) 2 1 d 2 1 d 2 ˆT λ 1 (λ 1 λ ) 1 m 1 (λ 1 ) 2 d 2 1T m 1 (1 + 2ρ 1 ) 2 The etry d 1 = w T 1 q 1 is the cosie of the agle φ 1 betwee the first eigevector w 1 ad the startig vector q 1 ; so we ca write (1 d 2 1)/d 2 1 = ta(φ 1 ) 2, which gives us the fial boud λ 1 θ 1 = max φ(p) φ(ĉ m 1)λ 1 (λ 1 λ ) ta(φ 1 ) 2 deg p<m T m 1 (1 + 2ρ 1 ) 2 where ρ 1 = (λ 1 λ 2 )/(λ 2 λ ) We otice a few thigs from this boud The rate of covergece (determied by T m 1 (1 + 2ρ 1 ) 2 ) is strictly better tha the rate of covergece for power iteratio, which makes sese sice we are usig strictly more iformatio i the Laczos iteratio tha we use i the power iteratio (we maximize over a m-dimesioal subspace rather tha a 1-dimesioal space) Also, the trick ivolved is a good oe: istead of reasoig about matrices, reaso about polyomials But, as with our aalysis of CG, this boud may be very pessimistic, sice it does ot take ito accout the distributio of eigevalues i [λ 2, λ ] For example, if may eigevalues of A cluster ear zero (as happes with certai discretizatios of compact operators, for istace), the we might get much better covergece tha idicated by a basic boud that works for ay distributio of eigevalues betwee [λ 2, λ ]