Math 577 Assignment 7

Math 577 Assignment 7 Thanks for Yu Cao 1. Solution. The linear system being solved is Ax = 0, where A is a (n 1 (n 1 matrix such that 2 1 1 2 1 A =......... 1 2 1 1 2 and x = (U 1, U 2,, U n 1. By the Jacobi method, A is split into two parts: A = D+E, where D is a diagonal matrix with the same diagonal elements as those of A, and E is a matrix whose diagonal elements are zero and whose off-diagonal elements are identical to those of A.Then Ax = 0 Dx = Ex x = D 1 Ex x = R J x, where R J = D 1 E Thus the weighted Jacobi method here can be rewrite as x q+1 = ωx J + (1 ωx q = ωr J x q + (1 ωx q = (ωr J + (1 ωix q = R ω x q where R ω = ωr J + (1 ωi (1 The exact solution x (if it can be got from the iteration is unchanged by the iteration, i.e., x = R ω x (2 By (1 and (2, e q+1 = R ω e q 1

where e q := x q x. By induction, e q = R q ωe 0, where e 0 = x 0 x, and x 0 is the initial guess. Notice that A is (n 1 (n 1, symmetric, positive definite, and x = 0 is the exact solution. Hence e q = x q. (i For ω = 1, see Figure 1 (a and Appendix errorfig.m. (ii For ω = 2/3, see Figure 1 (b and Appendix errorfig2.m. (iii No. Note that R ω = ωr J + (1 ωi = ω( D 1 E + (1 ωd 1 D = I ωd 1 (E + D = I ωd 1 A = I ω 2 A Hence λ(r ω = 1 ω 2 λ(a Note that the eigenvalues of A are λ k (A = 4 sin 2 ( kπ 2n, 1 k n 1 And the corresponding eigenvectors of A are exactly the n 1 initial guesses w k (1 k n 1 with n 1 components Then the eigenvalues of R ω are w k,j = sin( jkπ (1 j n 1 n λ k (R ω = 1 2ω sin 2 ( kπ 2n 1 k n 1, while the eigenvectors of R ω are the same as the eigenvectors of A. Expand the initial error by the eigenvectors of R ω After m iterations, n 1 e 0 = c k w k k=1 n 1 n 1 e m = Rω m e 0 = c k Rω m w k = c k λ m k w k k=1 Note that in this case which takes the eigenvector of A as the initial guess, e m = λ m k w k 2 k=1

when w k is the initial guess. Hence, for w k, the rate of convergence is depended on λ k = 1 2ω sin 2 ( kπ 2n (3 = kπ 1 ω(1 cos( n (4 The smaller is λ k, the faster is the convergence. By (4 one can see that for convergence of the iteration, 0 < ω 1. By (3, we have λ 1 = 1 2ω sin 2 ( π 2n 1 ωπ2 h 2 h = 1 2 n Thus when n is large, say n = 64 in this case, λ 1 is always close to 1, no matter what value of ω (0, 1] one takes. Therefore there is no optimal ω that reduces the error effectively for all w k. 2. (p.275 Ex. 35.4 Solution. (a First, the Givens rotation is represented by a n n orthogonal matrix G(i, j, θ n n = (g kl where the non-zero elements of G(i, j, θ n n is given by g kk = 1 g ii = g jj = c g ji = s g ij = s c = cos θ s = sin θ for k i, j Second, When a Givens rotation matrix, G(i, j, θ, multiplies another matrix, A, from the left, GA, only rows i and j of A are affected. Thus we restrict attention to the following problem. Given a and b, find c = cos θ and s = sin θ such that ( c s s c An obvious solution would be ( a b = r = a 2 + b 2 c = a/r s = b/r ( r 0 3

Here, we use the Givens rotation to deduce the QR factorization for the Hessenberg matrix H n in Algorithm 35.1. Let M k = G(k, k + 1, θ k (n+1 (n+1 where ( c(θk s(θ k s(θ k c(θ k ( hkk h k+1,k = ( h 2 kk + h2 k+1,k 0 Then M k M 2 M 1 Hn gives the upper-triangular matrix R n and H n = M T 1 M T 2 M T n R n = MR n gives the full QR factorization for H n. squares problem of Algorithm 35.1: Now we solve the least By Algorithm 11.2, Find y to minimize H n y b e 1 Algorithm I. Solving the least squares problem in Algorithm 35.1 1. Compute the full QR factorization H n = MR n. 2. Compute the vector M T ( b e 1. 3. Solve the upper-triangular system R n y = M T ( b e 1 for y. Here the QR factorization is implemented by Givens rotation described above. For counting the work for the algorithm in the box, we go into details of the QR factorization by Givens rotation. Algorithm II. Compute the QR factorization of a Hessenberg matrix by Givens rotation 1: for k = 1 : n do 2: r = h 2 kk + h2 k+1,k 3: c = h kk /r 4: s = h k+1,k [ /r ] c s 5: G k = [ s ] c [ ] Hk,1:n Hk,1:n 6: = G H k k+1,1:n H k+1,1:n 7: end for 4

The work in one loop is dominated by line 6. This is a matrix multiplication between a 2 2 and a 2 n matrices which cost 6n flops. Thus the whole work in Algorithm II is O(n 2. On the other hand, the work for Algorithm I is dominated by the cost of the QR factorization and thus is O(n 2. (b Note that H n 1 is a submatrix of Hn. One can get the QR factorization of H n from that of H n 1, which comes from the Givens rotation. If the problem for step n-1 has been solved, then when we deal with H n, the H n 1 block is already an upper-triangular matrix. We only need to change the last column of H n. Algorithm II can be modified as followings Algorithm III. Compute the QR factorization of a Hessenberg matrix by Givens rotation 1: for[ k = 1 : ] n 1 do [ ] hk,n hk,n 2: = G h k k+1,n h k+1,n 3: end for 4: r = h 2 nn + h 2 n+1,n 5: c = h nn /r 6: s = h n+1,n [ /r ] c s 7: G n = [ s ] c [ ] hn,n hn,n 8: = G h k n+1,n h n+1,n Note that G k (k = 1, 2,, n 1 had been constructed when factorized H n 1. The work dominated by the for loop in which the operation count is 3 2 (n 1 = 6(n 1 O(n. Hence the whole work involved in Algorithm I is improved to O(n. 3. (p. 275 Ex. 35.5 Solution. Suppose the initial guess is x 0 = m. Then we have A(x m = b Am. Modify the right-hand side b of Ax = b as b = b Am. Then the initial guess for A x = b is again x0 = 0, r 0 = b. Using the Algorithm 35.1 to solve A x = b(just replace b by b Am, let x n = x n + m. Things are done. 4. Show that φ(x is minimized for x = x n 1 + αp n 1 (with free choice of α when α = α n = r T n 1r n 1 /p T n 1Ap n 1. Solution. Note that φ(x = 1 2 xt Ax x T b 5

Denote x(α = x n 1 + αp n 1. For α that minimize φ(x(α, by the chain rule, we have d dα φ(x = 0 φ (x T d dα x(α = 0 ( 1 2 (AT x + Ax b T p n 1 = 0 (Ax b T p n 1 = 0 since A = A T Then (A(x n 1 + αp n 1 b T p n 1 = 0 (Ax n 1 b + αap n 1 T p n 1 = 0 ( r n 1 + αap n 1 T p n 1 = 0 p T n 1( r n 1 + αap n 1 = 0 α = pt n 1r n 1 p T n 1Ap n 1 (5 According to the algorithm 38.1, p n 1 = r n 1 + β n 1 p n 2. Then p T n 1r n 1 = r T n 1r n 1 + β n 1 p T n 2r n 1 (6 Since r n 1 K n 1 and p n 2 K n 1, we have p T n 2r n 1 = 0 (7 By (5(6(7 α = rt n 1r n 1 p T n 1Ap n 1 5. (p.302 Ex. 38.2 Solution. By the assumption, the 2-norm condition number κ = λ max /λ min = 24/1 = 24 Since ( n ( n e n A κ 1 24 1 2 = 2, e 0 A κ + 1 24 + 1 the maximum number of iterations required to achieve the bound 10 6 can be estimated by ( n 24 1 2 10 6 24 + 1 6

Thus n 10 6 log( 2 log ( 24 1 35.0396 24+1 Therefore, 36 steps of the CG iteration should be take to be sure of reducing the initial error e 0 A by a factor of 10 6. 6. Give an argument for the following statement: if the condition number κ is large but not too large, the CG iteration convergence to a specified tolerance can be expected in O( κ iterations. Solution. Suppose we wish to perform enough iterations to reduce the norm of the error by a factor of ε, i.e. Since e n A e 0 A ε ( n e n A κ 1 2 e 0 A κ + 1 the maximum number of iterations required to achieve the bound ε can be estimated by ( n κ 1 2 ε κ + 1 ( κ 1 n log log ( ε κ + 1 2 n log( ε 2 log ( log( ε κ 1 2 log ( log( ε 2 1 2 κ+1 κ 2 = 1 2 κ log( 2 ε = O( κ κ as κ The assumption that κ is not too large ensures the problem itself is of meaning. If the condition number κ is too large, the problem is ill-conditioned. Then even the relative error is within the specified tolerance, it can be useless. 7. (p.188 Ex. 24.1 (a True. If λ is an ew of A, then det(a λi = 0. Consequently, det((a µi (λ µi = det(a λi = 0, and thus λ µ is an ew of A µi. 7

(c True. If λ is an ew of real matrix A, then λ is a root of the real coefficient polynomial det(a xi, i.e. det(a λi = 0. The complex conjugate root theorem states that if P is a polynomial in one variable with real coefficients, and a + bi is a root of P with a and b real numbers, then its complex conjugate a bi is also a root of P. Hence det(a λi = 0, and therefore λ is also an ew of A. (d True. If λ is an ew of A and A is nonsingular, then λ 0. Assume that v is a non-zero vector such that Av = λv. Then A 1 (Av = (A 1 Av = v = λ 1 (λv = λ 1 (Av. Since A is nonsingular, and v 0, Av 0. Thus Av is an ev of A 1. (e False. Considering, A = ( 0 1 0 0 Obviously, all the ew s of A are zero, but A 0. (f True. If A is hermitian, i.e. A = A, then A = QΛQ = Q Λ sign(λq (8 where Λ and sign(λ denote the diagonal matices whose entries are the numbers λ j and sign(λ j, respectively. Since sign(λq is unitary whenever Q is unitary, (8 is an SVD of A, with the singular values equal to the diagonal entries of Λ, λ j. If desired, these numbers can be put into nonincreasing order by inserting suitable permutation matrices as factors in left-hand unitary matrix of (8, Q, and the right-hand unitary matrix, sign(λq. 8. (p.195 Ex 25.2Solution. (a In the case of linear convergence, e k+1 Ce k with C < 1 for all sufficiently large k. Without loss of generality, we can assume that e k+1 Ce k with C < 1 for all k 0. By induction, one can deduce that e n C n e 0, by which we have e n e 0 C n Denote ɛ = ɛ machine. Let C n ɛ we have n log C log ɛ n log ɛ log C 8

Since C < 1, C n is monotonically decreasing. For ensuring the accuracy O(ɛ machine, one only needs the low bound of n. Hence the steps needed are O(log(ɛ machine. Because the work of each step is O(1, the total work is O(log(ɛ machine. (b In this case e k+1 C(e k α with α > 1. Again assume that this is true for all k 0. Since (e k k=0 is convergent. without loss of generality, we can assume that 0 e 0 1. By induction, Then we have e n C 1+α+α2 + +α n 1 (e 0 αn e n e 0 M(e 0 αn where M = C 1+α+α2 + +α n 1 e 1 0 > 1 is a constant. Let M(e 0 αn ɛ, we have (e 0 αn ɛ M Then α n log(e 0 log( ɛ M α n log( ɛ M log(e 0 α n log(ɛ log(m log(ɛ log(e 0 log(e 0 ( log(ɛ n log(α log log(e 0 since log(e 0 < 0 Since α > 1, log(α > 0. Thus ( log(ɛ log log(e 0 n = log( log(ɛ log( log(e 0 log(α log(α = O(log( log(ɛ Consequently, the total work requirement in this case is O(log( log(ɛ machine. 9