1 Orthogonalisation in finite precision arithmetic

Transcription

1 1 Orthogonlistion in finite precision rithmetic We investigte the differences nd similrities between the following four wys to compute the QR-decomposition of given rectngulr mtrix A C m n in Mtlb: (CGS) (MGS) (HOU) (GIV) Clssicl Grm-Schmidt Modified Grm-Schmidt Householder reflections Givens rottions It turns out tht in finite precision rithmetic, the results my differ considerbly. 1.1 The QR-decomposition Let n m. A QR-decomposition of A C m n with columns 1,..., n is mtrix fctoristion A = QR (1) where Q is n m n mtrix whose columns q 1,..., q n re orthonorml nd hve the dditionl property tht spn{q 1,..., q l } = spn{ 1,..., l } for ll l {1,..., n}. (2) This implies tht R is n n n upper tringulr mtrix. A QR-decomposition exists for every A C m n. If A hs rnk n, there exists exctly one QR-decomposition of A in which the digonl entries of R re positive. See Theorem 7.2 in Trefethen & Bu. The computer generted results ˆQ nd ˆR of QR-decomposition of mtrix A re generlly not exct. Even though ˆR my be exctly upper tringulr, the entries of ˆR my well differ from those of R. Moreover, the columns of ˆQ re lmost surely not exctly orthogonl. This so-clled loss of orthogonlity results from finite precision rithmetic, the fct tht computers necessrily compute with only finitely mny numbers. 1.2 Tringulr Orthogonliztion: Clssicl versus Modified Grm Schmidt The Clssicl Grm-Schmidt (CGS) lgorithm is chrcterized by the following structure. Jorgen Pedersen Grm ( ) nd Erhrd Schmidt ( ) Assume tht the first l 1 columns q 1,..., q l 1 of the unitry fctor Q together with the (l 1) (l 1) leding principle submtrix of R re lredy known, then the l-th column q l of Q nd the l-th column of R re determined by the reltions r ll q l = l r 1l q 1 r 2l q 2... r l 1,l q l, where r jl = q j l (3) 1

2 for ll j {1,..., l 1}. Thus, j is orthogonlly projected on subspce of dimension l 1, being the spn of q 1,..., q l 1, nd the result is subtrcted from j. Equivlently one my sy tht j is projected onto the orthogonl complement of the spn of q 1,..., q l 1, which hs dimension m l + 1. Remrk: Formul (3) for this projection ssumes tht q 1..., q l 1 form n orthonorml bsis of its spn. Precisely this ssumption will not be fulfilled in finite precision rithmetic. This introduces, prt from rounding errors, lso modeling error. The Modified Grm-Schmidt (MGS) lgorithm is subtle vrint tht is in exct rithmetic equivlent with CGS. However, in finite precision rithmetic it generlly yields more ccurte results. Insted of (3), MGS evlutes r ll q l = (I q l 1 q l 1 )... (I q 2q 2)(I q 1 q 1) l. (4) The mthemticl equivlence with (3) follows from the fct tht if q 1 q 2 =, then (I q 2 q 2)(I q 1 q 1) = I q 1 q 1 q 2 q 2 + q 2 q 2q 1 q 1 = I q 1 q 1 q 2 q 2. This lso explins tht when implemented on computer, in which cse the computed columns of Q re not exctly orthonorml, the methods produce different results. Intuition: In expression (4), l is projected on spce of dimension m 1, the result is lso projected on spce of dimension m 1, nd so on: in totl there re l 1 consecutive projections on subspce of dimension m 1. Some intuition for the success of MGS is the following. Assume tht (I q 1 q1 ) l introduces n error due to finite precision rithmetic. This error is (smll) vector with components tht re rndomly distributed rndom over ll directions q 1,..., q m tht re going to be computed during the process. But m 1 of those m components will be projected to zero in the consecutive steps. Admittedly, pproximtely projected to zero, but this is better thn not doing nything with them t ll, s in (3). Importnt: Expressions of the form (I vv )w should never be implemented s (i) compute the m 2 entries of the m m mtrix I vv nd (ii) multiply with w. Much less expensive is to compute w v(v w), which does not involve ny mtrix Two equivlent orders of computtion in CGS nd MGS It is common prctice in both CGS nd MGS tht the column q l of Q is computed before ny computtions re done in which the columns l+1,..., n of A re involved, becuse these re not needed for the computtion of q l. Just becuse of tht, it is possible to choose nother order of computtion. Indeed, s soon s q 1 is computed, the complete first row of R cn lredy be computed. Moreover, ll columns 2,..., n of A cn be rid of their q 1 -components. For both CGS nd MGS this results in the sme vectors â 2,..., â n. Next, q 2 cn be computed by normlising â 2, nd â 3,..., â n cn be rid of their q 2 components. For CGS this results in vectors â C 3,..., âc n nd for MGS in â M 3,..., âm n, nd so on. Recll tht CGS nd MGS my produce different results only from the third vector onwrds. Remrk: Trefethen & Bu confuse the reder by giving the CGS lgorithm (Algorithm 7.1) in the usul order of computtion, nd MGS (Algorithm 8.1) in the lterntive order. This suggests tht this spect explins the better results of MGS in comprison with CGS, which is flse. The rel gin lies in the difference between (3) nd (4)! 2

3 1.2.2 Exercise () Implement Algorithms 7.1 nd 8.1 in Mtlb s [Q,R]=CGS(A) nd [Q,R]=MGS(A). (b) Give implementtions of CGS nd MGS in which the other order of computtion is used. (c) Verify by exmple tht both versions of CGS produce the sme result. (d) Verify by exmple tht both versions of MGS produce the sme result. Such verifictions re best done with smll exmples in full precision, mening, using Mtlb s formt long. You my wish to store output using the diry commnd of Mtlb Exercise In Mtlb, the commnd A=hilb(n) genertes the Hilbert mtrix of dimensions n n. Use both CGS nd MGS to compute QR-decomposition of the n n Hilbert mtrix for n {4, 5,..., 1}. For ech QR-decomposition, compute using Mtlb the norm I Q Q, which equls zero if nd only if Q hs orthonorml columns. Use the semilogy commnd nd plot both the 7-vectors with norms for CGS nd MGS in logrithmic scle. Use hold to get both grphs in the sme plot. With semilogy(v, -*r ) you cn drw the grph for v in red nd with sterisks, wheres with semilogy(v, -o ) it is in blue (defult) with circles. With legend you cn mke legend nd reposition it using the mouse. See lso grid title xlbel ylbel to pimp the picture even more. Export nd store it s jpg-file. Keep the vribles in memory, the ones from MGS you will use gin in Exercise 1.4.2! 1.3 Tringulr Orthogonliztion versus Orthogonl Tringulriztion As rgued on pge 61 in Trefethen & Bu you cn interpret the orthogonliztion of A using Grm-Schmidt s tringulr orthogonliztion. Indeed, s soon s q 1 is determined, the first row of R cn be computed nd ll columns fo A cn be ridded of their q 1 -components. This corresponds to right-multipliction of A by n upper tringulr mtrix, nd results in n itertive process, A = A, A 1 = A R 1, A 2 = A 1 R 2,..., A n = A n 1 R n, (5) where R j is the mtrix tht removes the q j -component from columns j +1,..., n of the mtrix it cts upon (from the right). Thus, fter n steps, the unitry fctor remins: A n = AR 1 R 2 R n = Q, nd hence, R 1 R 2... R n = R 1. (6) Alterntively, one my try to multiply the mtrix A itertively from the left with unitry mtrix until the upper tringulr fctor remins: nd thus,  = A,  1 = Q 1   2 = Q 2  1...,  n = Q n  n 1, (7) Q n Q 2 Q 1 A = R, nd hence, Q n Q 2 Q 1 = Q. (8) This ltter strtegy will generlly produce better results in finite precision rithmetic. To understnd this, consider the instructive (though not very relistic) sitution in which A 3

4 itself contins errors of some kind, but tht for some mysterious reson in (6) nd (8) ll computtions re performed exctly. Writing the inexct strt mtrix s A + E, the excte mtrix A plus n error mtrix E, then with (6) nd (8) we find tht (A + E)R 1 R 2 R n = AR 1 + ER 1 nd Q n Q 2 Q 1 (A + E) = Q A + Q E. (9) Mesured in unitrily invrint norm (such s 2 nd F ) we know however tht Q E E (1) wheres ER 1 cn in principle be much lrger thn E. Indeed, ER 1 is the solution X of the upper tringulr system XR = E, whose solution cn be much lrger thn the dt due to smll pivots! Remrk: The ssumption tht ll opertions on A + E re performed exctly is unrelistic, nd the negtive effects of tringulr orthogonliztion will in prctice itertively ccumulte! Exercise Let A=hilb(8) nd choose E=.1*rnd(8), mtrix with smll entries compred to those of A. Compute using the Mtlb commnd [Q,R]=qr(A) QR-decomposition of A. Mtlb uses for this the method of the next section. () How orthonorml is this Q in comprison with the one computed in Exercise 1.2.3? Use now the Mtlb commndo [QE,RE]=qr(A+E) to compute QR-decomposition of A + E. Motivted by the result in () we will pretend tht this QR-decomposition is the exct QRdecomposition of A + E, nd thus, tht its fctors re the ones from (9). (b) Compute norm(qe *E)/norm(E) en norm(e*inv(re))/norm(e). Use diry to store your results. Do they confirm the intuition given bove? Add your comments to your diry-file (simply by opening it in text editor). 1.4 Orthogonl Tringulriztion: Householder reflections It is not difficult to find the mtrix of reflection S in hyperplne V, tht mps given x C m on x e 1, where e 1 is the first cnonicl bsis vector of C m. Indeed, the difference v = x x e 1 should be orthogonl to the plne V of reflection, nd thus P (x) = (I vv v v )x = x v (v x) (v v) en S(x) = (I 2 vv v v )x = x 2v (v x) (v v) (11) re respectively equl to the orthogonl projection of x onto V nd the reflection of x in V. See lso Exercise 6.1 in Trefethen & Bu, which introduces reflections. Given ny m n mtrix A with 1 = Ae 1 we cn use (11) to compute the (unitry) mtrix H 1 for which 1... b b 2n H 1 A =.... (12) b m2... b mn 4

5 Similrly, we cn lso determine the mtrix of the reflection tht mps 1 on 1 e 1, for which the hyperplne of reflection is orthogonl to 1 + x 1. This second reflection comes in hndy if 1 en 1 e 1 re lmost equl: s is well known, subtrcting two lmost equl numbers leds to loss of significnt digits, so-clled cncelltion errors. Alston Householder ( ) For this reson, Alston Householder proposed to use the sign of the first entry of x to determine whether to reflect x on x e 1 or on x e 1 : choose v = x + sgn(e 1 x) x e 1 nd reflect in v. Numericlly this is the stble choice. For the purpose of computing QR-decomposition of mtrix A using itertive left-multipliction with unitry mtrices, this mkes no difference. We cn now continue to find the mtix H 2 tht reflects the first column of the (m 1) (n 1) bottom right submtrix of H 1 A onto multiple of the first cnonicl bsis vector, nd so on, to compute QR-decomposition of A. The unitry mtrix Q cn then be explicitly computed s product of ll reflection mtrices. One my lso choose to simply store ll norml vectors to the reflection plnes. It depends on the ctul ppliction of the QR-decomposition if you relly need the fctor Q explicitly. See Algorithms 1.2 nd 1.3 in Trefethen & Bu Exercise Implement QR-decomposition using Householder reflections s [Q,R]=House(A). Compute the unitry mtrix Q by hving ech reflection ct on both A nd I until A hs been tringulrized. Indeed, t tht moment, I hs been trnsformed into Q Exercise Use your code House to compute QR-decomposition of the Hilbert mtrices of size n n for n {4, 5,..., 1}. Agin compute I Q Q. Just s in Exercise 1.2.3, mke grph, but now compre MGS with Householder QR, nd store it s jpg-file. Agin, do not throw wy your computtionl results just yet. 1.5 Orthogonl Tringulriztion: Givens rottions Aprt from reflections, lso rottions cn be used s unitry trnsformtions with which mtrix A cn be tringulrized. Given nonzero vector in R 2 it is esy to write down the orthogonl mtrix of the rottion bout the origin tht mps this vector on multiple of e 1, R(, b) [ b ] [ ] = 2 + b 2 where R(, b) = [ b 2 b b ]. (13) 5

6 Note: From the bove we cn lredy conclude tht [ ] [ A = = R(, b) 2 + b 2 b ] 1 = UΣV (14) which is singulr vlue decomposition of the 2 1 mtrix ( b)! A vector x = (x 1, x 2, x 3, x 4 ) R 4 cn be isometriclly mpped onto multiple of e 1 by product of three rottions. The first is in the (x 3, x 4 )-plne, the second in the (x 2, x 3 )-plne, nd the third in the (x 1, x 2 )-plne. Schemticlly, x 1 x 2 x 3 x 4 R 34(x 3,x 4) x 1 x 2 y 3 R 23(x 2,y 3) x 1 y 2 R 12(x 1,y 2) y 1. (15) Next, we show how to employ such rottions to bring 4 3 mtrix in upper tringulr form. This we do column-wise s follows. Apply the three rottions mpping the first column of X onto multiple of e 1 ; X = R 34(, ) R 23(, ) R 12(, ) = Y. The product Y = R 12 R 23 R 34 X hs three zeros below (1, 1). Note in prticulr tht the second nd third column of Y re generlly not the sme s those of X! Apply two rottions to crete zeros on the two positions below (2, 2) vn Y ; Y = R 34(, ) R 23(, ) = Z. Observe tht this cn be done without destroying previously introduced zero entries. Apply the rottion tht introduces nother zero entry, t position (3, 4); Z = R 34(, ) = U. The finl result U, fter pplying six rottions, is upper tringulr. Wllce Givens ( ) 6

7 A plne rottion in the context of computing QR-decomposition is nmed fter Wllce Givens. In de context of similrity trnsforms, they re clled fter Crl Jcobi Exercise Investigve the commnd givens nd use it to write code [Q,R]=PlneRots(A) tht computes QR-decomposition of A using Givens rottions. In this implementtion, compute Q by pplying the rottions to both A nd I until A is tringulrized Exercise Use your code PlneRots to compute QR-decomposition of the Hilbert mtrices of size n n for n {4, 5,..., 1}. Compute I Q Q. Just s in Exercises nd 1.4.2, mke grph compring Householder QR nd Givens QR. 1.6 Hnding in Hnd in the following the m-files of both vrints of CGS; the m-files vn both vrints of MGS; the m-file for House; the m-file for PlneRots; the jpg s of your figures for Exercises nd nd the diry prts for Exercises nd 1.3.1, in one file, plese. Thus, ten files in totl. Plese send them s seprte files in one e-mil (i.e. without zipping) to Jn: J.H.Westerdiep@uv.nl Dedline for this ssignment is Sundy October 1, 23:59 hrs. Wrning: This ssignment not only counts towrds your finl grde, it lso decides bout pssing for the course. Therefore, ech student hs to hnd in his or her work by him/herself. Copying (or cosmeticlly dpting with the purpose of hiding plgirism) codes or prts of codes of fellow students is strictly prohibited nd my led to hering by the FNWI Plgirism Bord. However, coopertion with fellow students on globl level is llowed nd encourged. In tht cse it is dviced (nd good prctice) to explicitly stte in your work with whom you hve cooperted, s to explin possible superficil similrity between your work. 7