Basic Iterative Methods. Basic Iterative Methods

Transcription

1 Abel s heorem: he roots of a polyomial with degree greater tha or equal to 5 ad arbitrary coefficiets caot be foud with a fiite umber of operatios usig additio, subtractio, multiplicatio, divisio, ad extractio of roots. [he roots of some special polyomials of ay order like (x ) = 0 ca be foud.] As a cosequece: Ay geeral methods for fidig eigevalues are iterative 0. Power Method: If A is semi-simple the ay vector q may be writte q = c x + c 2 x 2 + +c x, where the c j are costats ad the x j are the m m m m eigevectors. We the have A q c xc22x2 c x If c 0 ad > j, < j, the m m m cx lim A q ad lim[ A q] k / [ A q] m k m m where [A m q] k refers to ay o-zero elemet of A m q. Repeatedly iteratig, we ultimately obtai the largest eigevalue ad eigevector. Some issues: () Covergece is oly liear (ot fast) (2) We oly get the largest eigevalue ( fixable by deflatio) (3) Whe the two largest magitude eigevalues have the same magitude, the method does ot coverge 0.2

2 Power Method: Example : Cosider the matrix 9 A 2 We first peek at the aswer usig MALAB: >> A = [9 ; 2] >> [V, D] = eig(a) >> format log, D MALAB has more accuracy tha ormally show We ext carry out the power iteratio >> eps = ; = 0; Iitialize stop criterio, operatio cout >> q = [ ]'; Iitialize the iteratio vector q = A*q; lambda = q()/q(), q = q/max(abs(q)); eps = orm(q - q); q = q; = + ed Carry out the iteratio We see that covergece is reasoable but ot great 0.3 Power Method: Example 2: Cosider the matrix A 0 We carry out the iteratio oce agai: >> A = [0.99 0; 0 ] >> eps = ; = 0; Iitialize stop criterio, operatio cout >> q = [ ]'; Iitialize the iteratio vector q = A*q; [C, I] = max(abs(q)); lambda = q(i)/q(i), q = q/c; eps = orm(q - q); q = q; = + ed Carry out the iteratio Covergece is much slower! 0.4 2

3 Power Method: Example 3: Cosider the matrix A We first peek at the aswer usig MALAB: >> A = [ ; - 9 2; -4-2] >> [V, D] = eig(a) he largest magitude eigevalue is real We ext carry out the power iteratio >> eps = ; = 0; Iitialize stop criterio, operatio cout >> q = [ ]'; Iitialize the iteratio vector q = A*q; lambda = q()/q(), q = q/max(abs(q)); eps = orm(q - q); q = q; = + ed Carry out the iteratio No serious problem i this case 0.5 Power Method: Example 4: But with a small chage A We first peek at the aswer usig MALAB: >> A = [ ; - 3 2; -4-2] >> [V, D] = eig(a) he largest magitude eigevalues have the same magitude We ext carry out the power iteratio >> eps = ; = 0; Iitialize stop criterio, operatio cout >> q = [ ]'; Iitialize the iteratio vector q = A*q; lambda = q()/q(), q = q/max(abs(q)); eps = orm(q - q); q = q; = + ed Carry out the iteratio Ad there is o covergece! 0.6 3

4 Shift-ad-Ivert Method: It is better to work backwards. Suppose you have a good guess for, the the eigevalues of A I are, 2,,, ad the largest eigevalue of (A I) will be ( ) ad will quickly coverge. he eigevectors do ot chage! Note: I practice, we do ot ivert, but solve (A I)q m+ = q m before rescalig (e.g., usig LU decompositio) 9 Example 5: We cosider agai A 2 ad guess = 9 We first fid (A I) ad peek at its eigevectors ad eigevalues: >> A = [0 ; -7]; Ai = iv(a), [V, D] = eig(ai) Note the large differece betwee eigevalues! more tha a factor of 50!! 0.7 Shift-ad-Ivert Method: 9 Example 5 (cotiued): We cosider agai A ad guess = 9 2 We first fid (A I) : ad peek at its eigevectors ad eigevalues >> A = [0 ; -7]; Ai = iv(a), [V, D] = eig(ai) Note the large differece betwee eigevalues! We ext carry out the power iteratio >> eps = ; = 0; q = [ ]'; Iitializatio q = Ai*q; lambda = 9 + q()/q(), q = q/max(abs(q)); (q eps = orm(q - q); q = q; = + ed Carry out the iteratio Oly a small umber of iteratios are eeded! ad we ca update ad do eve better 0.8 4

5 Rayleigh Quotiet: Cosider the equatio Aq = q. If q is ot a eigevector, the there is o solutio for, but the least square residual Aq q is miimized if we choose = q Aq/ q q, which is the Rayleigh quotiet 9 Example 6: We cosider agai A ad guess first = 9 2 We first iitialize A ad >> A = [9 ; 2], rho = Rayleigh Quotiet: Example 6 (cotiued: We first iitialize A ad >> A = [9 ; 2], rho = 9 We ext carry out the power iteratio >> eps = ; = 0; q = [ ]'; Iitializatio As = A - rho*eye(2); q = As\q; lambda = rho + q()/q(), rho = q'*a*q/(q'*q), [M, I] = max(abs(q)); q = q*sig(q(i))/m; eps = orm(q - q), q = q; = + ed Carry out the iteratio i Covergece is very rapid! A importat issue is that we must solve for q o each iteratio which is a 3 operatio

6 Matrix Simplificatio Hesseberg ad ridiagoal Forms: he basic idea: We caot use Householder matrices to directly carry out a Schur (similarity) decompositio. If left multiplyig A by U makes all the colum elemets below the diagoal trasformed to zero, i.e, the right multiplyig U A by U makes them o-zero agai. BU: You ca zero out all the elemets i the colums below the diagoal except the topmost oes. he matrix the has the form show below. his form is upper Hesseberg. his form becomes tridiagoal for ormal matrices. Ay iterative operatios are ow O( 2 ) which is a big wi! * * * * * 0 0 * * * * * * A 0 * for ormal matrices 0 * 0 * * 0 0 * * 0 0 * * 0. Matrix Simplificatio Hesseberg ad ridiagoal Forms: Example (A asymmetric matrix): >> A = rad(4) >> [P, H] = hess(a) >> Check = P*H*P' Example 2 (A symmetric matrix): >> B = A*A' >> [PB, HB] = hess(a) Note that this is tridiagoal! 0.2 6

7 Statemet of the Algorithm: QR Algorithm he goal: Fid a similarity trasformatio that reduces a arbitrary matrix to Schur form. he iterative procedure: A 0 = A (begi the iteratio) U m R m = A m (make a QR decompositio, m =, 2,, N ) A m = R m U m (Reverse U m ad R m to obtai a ew matrix) QR iteratio theorem: UU U A UU U lim N 2 N 2 N N where is the upper triagular Schur matrix Easy to state easy to implemet Not so easy to prove! 0.3 QR Algorithm Partial Proof: (For semi-simple matrices with > 2 > > ) We will proceed iteratively. At the first stage will cosider power iteratios. We let U equal a r matrix ad let U 0 = [e e 2 e r ], where the ej are the colum vectors whose j-th elemet is zero ad all others are. We ow cosider the followig iteratio Z m = AU m (Multiply by A) R m U m = Z m (Make a QR decompositio) First step: We first suppose that r =. I that case, we have implemeted the power algorithm that we described previously, where we use e as the startig vector ad we use the 2-orm to scale. We ote that R m is just a scalar (a matrix) ad equals the 2-orm of Z m. As log as c 0ithe sum e, where the j c jv j v j are the eigevectors, ormalized to we fid Ae c v, U c v, j j j j j j j j c j j j v j where 0.4 7

8 QR Algorithm Partial Proof (cotiued): For ay > 0, it is possible to fid a umber of iteratios N such the N -th iteratio of the algorithm produces v + r, where r <. We choose N large eough so that subsequet iteratios of the algorithm are ot affected. So, we may write u v as the value to which we coverge. Mathematically, this is a fudge! o be rigorous, we would have to carry r through the calculatio. hat would itroduce substatial complicatio ad little additioal isight. Secod step: We ext suppose that r = 2. Startig with [e e 2 ], where we assume v 2 e 2 0. After N iteratios, we have U N u x2 x2 c j 2 jv j c 22, where ad 0 We ow fid, recallig that QR decompositio is equivalet to Gram- Schmidt orthogoalizatio 0.5 Z R QR Algorithm u c v, U u c v v v v, N j 2 j j j N N j2 2 j j 2 j j c 2 2 jj v j v j N N 0 2 N he ext iteratio gives us Z, where ormalizes the secod colum 2 u c v v v v N 2 N 2 j2 2 j j 2 j j j 2 U u c v v v v where ormalizes the secod colum N 2 N 2 j2 2 j j 2 j j N 2 he Gram-Schmidt process elimiates growth proportioal to! Cotiuig with N 2 additioal steps, we coverge to,, 0.6 8

9 Z R QR Algorithm U v v v2 vv2u, where u2 2 v2 vv2u N N N N 2 2 N N 2 u u v v u, u u, hird Step: We ext move to r = 3, 4,, addig extra iteratios as eeded. Whe we coverge to a acceptable value with r =, the R = (the Schur upper triagular matrix) ad R = U AU. Note: We solved the problem, assumig c 0, c 22 0, Sice the e j spa the etire vector space, this assumptio is ot eeded as log as we do colum pivotig, which is eeded i ay case for stability. Fial Observatio: O the (k )-th iteratio with r =, we have ad we also have R U AU U AU U U R k k k k k k k k 0.7 Fial Observatio: O the (k )-th iteratio with r =, we have ad we also have QR Algorithm U AU U AU U U R k k k k k k k k U AU U AU U U R U U k k k k k k k k k k his is precisely what the QR algorithm does Remark : Much of the complexity i the stadard discussios is aimed at determiig the rate of covergece a issue that we bypassed. Remark 2: Covergece is oly liear. Back iteratio with clever shiftig dramatically speeds the algorithm. he best approach depeds o the matrix structure (real or complex, symmetric or arbitrary, ot sparse or sparse )

10 QR Algorithm Example: We cosider oce agai 9 A 2 Usig MALAB: >> format log So we ca see covergece >> A = [9 ; 2], k = A; U = eye(2); Uk = U; = 0; Iitializatios >> while abs(k(2,)) > e-2 [Uk R] = qr(k); U = U*Uk; k = R*Uk, = + ed Iterate usig the lower diagoal elemet value as the stoppig criterio Simple, but ot very fast! 0.9 MALAB Algorithms So what does MALAB do?: For full matrices: >> doc eig For sparse matrices: >> doc eigs It is quite sophisticated! he bottom lie: () Use caed routies BU (2) Kow what you are doig!

11 SVD Revisited We recall that the sigular value factorizatio is give by (slide 7.7): heorem: Let A be a m matrix of rak r. he, A ca be expressed as a product A = UΣV, where U ad V are respectively m m ad orthogoal (uitary) matrices of rak r, ad Σ is a o-square diagoal m matrix of rak r, 2 Σ, 2 r 0 r 0 We see that AA UΣV VΣ U UΣΣ U ad A A VΣ U UΣV VΣΣ V so that fidig the SVD becomes a eigevalue problem! 0.2