2.5 Iterative Improvement of a Solution to Linear Equations

Transcription

1 2.5 Iteratve Improvement of a Soluton to Lnear Equatons 47 Dahlqust, G., and Bjorck, A. 1974, Numercal Methods (Englewood Clffs, NJ: Prentce-Hall), Example 5.4.3, p Ralston, A., and Rabnowtz, P. 1978, A Frst Course n Numercal Analyss, 2nd ed. (New York: McGraw-Hll), Wlknson, J.H., and Rensch, C. 1971, Lnear Algebra, vol. IIofHandbook for Automatc Computaton (New York: Sprnger-Verlag), Chapter I/6. [1] Golub, G.H., and Van Loan, C.F. 1989, Matrx Computatons, 2nd ed. (Baltmore: Johns Hopkns Unversty Press), Iteratve Improvement of a Soluton to Lnear Equatons Obvously t s not easy to obtan greater precson for the soluton of a lnear set than the precson of your computer s floatng-pont word. Unfortunately, for large sets of lnear equatons, t s not always easy to obtan precson equal to, or even comparable to, the computer s lmt. In drect methods of soluton, roundoff errors accumulate, and they are magnfed to the extent that your matrx s close to sngular. You can easly lose two or three sgnfcant fgures for matrces whch (you thought) were far from sngular. If ths happens to you, there s a neat trck to restore the full machne precson, called teratve mprovement of the soluton. The theory s very straghtforward (see Fgure 2.5.1): Suppose that a vector x s the exact soluton of the lnear set A x = b (2.5.1) You don t, however, know x. You only know some slghtly wrong soluton x + δx, where δx s the unknown error. When multpledby the matrx A, your slghtlywrong solutongves a product slghtlydscrepant from the desred rght-handsde b, namely Subtractng (2.5.1) from (2.5.2) gves A (x + δx) =b+δb (2.5.2) A δx = δb (2.5.3) But (2.5.2) can also be solved, trvally, for δb. Substtutng ths nto (2.5.3) gves A δx = A (x + δx) b (2.5.4) In ths equaton, the whole rght-hand sde s known, snce x + δx s the wrong soluton that you want to mprove. It s essental to calculate the rght-hand sde n double precson, snce there wll be a lot of cancellaton n the subtracton of b. Then, we need only solve (2.5.4) for the error δx, then subtract ths from the wrong soluton to get an mproved soluton. An mportant extra beneft occurs f we obtaned the orgnal soluton by LU decomposton. In ths case we already have the LU decomposed form of A, and all we need do to solve (2.5.4) s compute the rght-hand sde and backsubsttute! The code to do all ths s concse and straghtforward:

2 48 Chapter 2. Soluton of Lnear Algebrac Equatons A C δx x + δx x A 1 b b + δb Fgure Iteratve mprovement of the soluton to A x = b. The frst guess x + δx s multpled by A to produce b + δb. The known vector b s subtracted, gvng δb. The lnear set wth ths rght-hand sde s nverted, gvng δx. Ths s subtracted from the frst guess gvng an mprovedsolutonx. SUBROUTINE mprove(a,alud,n,np,ndx,b,x) INTEGER n,np,ndx(n),nmax REAL a(np,np),alud(np,np),b(n),x(n) PARAMETER (NMAX=500) Maxmum antcpated value of n. USES lubksb Improves a soluton vector x(1:n) of the lnear set of equatons A X = B. The matrx a(1:n,1:n), and the vectors b(1:n) and x(1:n) are nput, as s the dmenson n. Also nput s alud, thelu decomposton of a as returned by ludcmp, and the vector ndx also returned by that routne. On output, only x(1:n) s modfed, to an mproved set of values. INTEGER,j REAL r(nmax) DOUBLE PRECISION sdp do 12 =1,n Calculate the rght-hand sde, accumulatng the resdual n double precson. sdp=-b() do 11 j=1,n sdp=sdp+dble(a(,j))*dble(x(j)) enddo 11 r()=sdp enddo 12 call lubksb(alud,n,np,ndx,r) Solve for the error term, do 13 =1,n and subtract t from the old soluton. x()=x()-r() enddo 13 return END You should note that the routne ludcmp n 2.3 destroys the nput matrx as t LU decomposes t. Snce teratve mprovement requres both the orgnal matrx and ts LU decomposton, you wll need to copy A before callng ludcmp. Lkewse lubksb destroys b n obtanng x, so make a copy of b also. If you don t mnd ths extra storage, teratve mprovement s hghly recommended: It s a process of order only N 2 operatons (multply vector by matrx, and backsubsttute see dscusson followng equaton 2.3.7); t never hurts; and t can really gve you your money s worth f t saves an otherwse runed soluton on whch you have already spent of order N 3 operatons. δb

3 2.5 Iteratve Improvement of a Soluton to Lnear Equatons 49 You can call mprove several tmes n successon f you want. Unless you are startng qute far from the true soluton, one call s generally enough; but a second call to verfy convergence can be reassurng. More on Iteratve Improvement It s llumnatng (and wll be useful later n the book) to gve a somewhat more sold analytcal foundatonfor equaton (2.5.4), and also to gve some addtonal results. Implct n the prevous dscusson was the noton that the soluton vector x + δx has an error term; but we neglected the fact that the LU decomposton of A s tself not exact. A dfferent analytcal approach starts wth some matrx B 0 that s assumed to be an approxmate nverse of the matrx A, sothatb 0 As approxmately the dentty matrx 1. Defne the resdual matrx R of B 0 as R 1 B 0 A (2.5.5) whch s supposed to be small (we wll be more precse below). Note that therefore B 0 A = 1 R (2.5.6) Next consder the followng formal manpulaton: A 1 = A 1 (B B 0)=(A 1 B0 ) B 0 =(B 0 A) 1 B 0 =(1 R) 1 B (2.5.7) 0 =(1+R+R 2 +R 3 + ) B 0 We can defne the nth partal sum of the last expresson by B n (1 + R + +R n ) B 0 (2.5.8) so that B A 1, f the lmt exsts. It now s straghtforward to verfy that equaton (2.5.8) satsfes some nterestng recurrence relatons. As regards solvng A x = b, wherexand b are vectors, defne x n B n b (2.5.9) Then t s easy to show that x n+1 = x n + B 0 (b A x n) (2.5.10) Ths s mmedately recognzable as equaton (2.5.4), wth δx = x n+1 x n, and wth B 0 takng the role of A 1. We see, therefore, that equaton (2.5.4) does not requre that the LU decomposton of A be exact, but only that the mpled resdual R be small. In rough terms, f the resdual s smaller than the square root of your computer s roundoff error, then after one applcaton of equaton (2.5.10) (that s, gong from x 0 B 0 b to x 1) the frst neglected term, of order R 2, wll be smaller than the roundoff error. Equaton (2.5.10), lke equaton (2.5.4), moreover, can be appled more than once, snce t uses only B 0, and not any of the hgher B s. A much more surprsng recurrence whch follows from equaton (2.5.8) s one that more than doubles the order n at each stage: B 2n+1 =2B n B n A B n n=0,1,3,7,... (2.5.11) Repeated applcaton of equaton (2.5.11), from a sutable startng matrx B 0, converges quadratcally to the unknown nverse matrx A 1 (see 9.4 for the defnton of quadratcally ). Equaton (2.5.11) goes by varous names, ncludng Schultz s Method and Hotellng s Method; see Pan and Ref[1] for references. In fact, equaton (2.5.11) s smply the teratve Newton-Raphson method of root-fndng ( 9.4) appled to matrx nverson. Before you get too excted about equaton (2.5.11), however, you should notce that t nvolves two full matrx multplcatons at each teraton. Each matrx multplcaton nvolves N 3 adds and multples. But we already saw n that drect nverson of A requres only N 3 adds and N 3 multples n toto. Equaton (2.5.11) s therefore practcal only when specal crcumstances allow t to be evaluated much more rapdly than s the case for general matrces. We wll meet such crcumstances later, n

4 50 Chapter 2. Soluton of Lnear Algebrac Equatons In the sprt of delayed gratfcaton, let us nevertheless pursue the two related ssues: When does the seres n equaton (2.5.7) converge; and what s a sutable ntal guess B 0 (f, for example, an ntal LU decomposton s not feasble)? We can defne the norm of a matrx as the largest amplfcaton of length that t s able to nduce on a vector, R max v 0 R v v (2.5.12) If we let equaton (2.5.7) act on some arbtrary rght-hand sde b, as one wants a matrx nverse to do, t s obvous that a suffcent condton for convergence s R < 1 (2.5.13) Pan and Ref [1] pont out that a sutable ntal guess for B 0 s any suffcently small constant ɛ tmes the matrx transpose of A, thats, B 0=ɛA T or R = 1 ɛa T A (2.5.14) To see why ths s so nvolves concepts from Chapter 11; we gve here only the brefest sketch: A T A s a symmetrc, postve defnte matrx, so t has real, postve egenvalues. In ts dagonal representaton, R takes the form R = dag(1 ɛλ 1, 1 ɛλ 2,...,1 ɛλ N) (2.5.15) where all the λ s are postve. Evdently any ɛ satsfyng 0 <ɛ<2/(max λ ) wll gve R < 1. It s not dffcult to show that the optmal choce for ɛ, gvng the most rapd convergence for equaton (2.5.11), s ɛ =2/(max λ +mnλ ) (2.5.16) Rarely does one know the egenvalues of A T A n equaton (2.5.16). Pan and Ref derve several nterestng bounds, whch are computable drectly from A. The followng choces guarantee the convergence of B n as n, / /( ) ɛ 1 a 2 jk or ɛ 1 max a j max a j (2.5.17) j j,k The latter expresson s truly a remarkable formula, whch Pan and Ref derve by notng that the vector norm n equaton (2.5.12) need not be the usual L 2 norm, but can nstead be ether the L (max) norm, or the L 1 (absolute value) norm. See ther work for detals. Another approach, wth whch we have had some success, s to estmate the largest egenvalue statstcally, by calculatng s A v 2 for several unt vector v s wth randomly chosen drectons n N-space. The largest egenvalue λ can then be bounded by the maxmum of 2maxs and 2NVar(s )/µ(s ), where Var and µ denote the sample varance and mean, respectvely. CITED REFERENCES AND FURTHER READING: Johnson, L.W., and Ress, R.D. 1982, Numercal Analyss, 2nd ed. (Readng, MA: Addson- Wesley), 2.3.4, p. 55. Golub, G.H., and Van Loan, C.F. 1989, Matrx Computatons, 2nd ed. (Baltmore: Johns Hopkns Unversty Press), p. 74. Dahlqust, G., and Bjorck, A. 1974, Numercal Methods (Englewood Clffs, NJ: Prentce-Hall), 5.5.6, p Forsythe, G.E., and Moler, C.B. 1967, Computer Soluton of Lnear Algebrac Systems (Englewood Clffs, NJ: Prentce-Hall), Chapter 13. Ralston, A., and Rabnowtz, P. 1978, A Frst Course n Numercal Analyss, 2nd ed. (New York: McGraw-Hll), 9.5, p Pan, V., and Ref, J. 1985, n Proceedngs of the Seventeenth Annual ACM Symposum on Theory of Computng (New York: Assocaton for Computng Machnery). [1] j

5 2.6 Sngular Value Decomposton Sngular Value Decomposton There exsts a very powerful set of technques for dealng wth sets of equatons or matrces that are ether sngular or else numercally very close to sngular. In many cases where Gaussan elmnaton and LU decomposton fal to gve satsfactory results, ths set of technques, known as sngular value decomposton, orsvd, wll dagnose for you precsely what the problem s. In some cases, SVD wll not only dagnose the problem, t wll also solve t, n the sense of gvng you a useful numercal answer, although, as we shall see, not necessarly the answer that you thought you should get. SVD s also the method of choce for solvng most lnear least-squares problems. We wll outlne the relevant theory n ths secton, but defer detaled dscusson of the use of SVD n ths applcaton to Chapter 15, whose subject s the parametrc modelng of data. SVD methods are based on the followngtheorem of lnear algebra, whose proof s beyond our scope: Any M N matrx A whose number of rows M s greater than or equal to ts number of columns N, can be wrtten as the product of an M N column-orthogonal matrx U, ann Ndagonal matrx W wth postve or zero elements (the sngular values), and the transpose of an N N orthogonal matrx V. The varous shapes of these matrces wll be made clearer by the followng tableau: A = U w 1 w 2 w N V T (2.6.1) The matrces U and V are each orthogonal n the sense that ther columns are orthonormal, M U k U n = δ kn =1 N V jk V jn = δ kn j=1 1 k N 1 n N 1 k N 1 n N (2.6.2) (2.6.3)