Sherman-Morrison-Woodbury

Transcription

1 Week 5: Wednesday, Sep 23 Sherman-Mrison-Woodbury The Sherman-Mrison fmula describes the solution of A+uv T when there is already a factization f A. An easy way to derive the fmula is through block Gaussian elimination. In der to compute the product (A+uv T x, we would usually first compute ξ v T x and then compute (A+uv T x Ax+uξ. So we can write (A + uv T x b in terms of an extended linear system [ ] [ ] [ ] A u x b v T. 1 ξ 0 We can fact the matrix in this extended system as [ ] [ ] [ ] A u I 0 A u v T 1 v T A v T A 1, u apply fward substitution with the block lower triangular fact, y b, η v T A 1 y, and apply backward substitution with the block upper triangular fact, ξ ( 1 v T A 1 u 1 η x A 1 (y uξ. If we put all the algebra together, we find ] x [A 1 A 1 uv T A 1 b, 1 + v T A 1 u (A + uv T 1 A 1 A 1 uv T A v T A 1 u. This last fmula is usually called the Sherman-Mrison fmula. The Sherman-Mrison-Woodbury fmula is the generalization to a rank k modification to A: (A + UV T 1 A 1 A 1 U(I + V T A 1 U 1 V T A 1.

2 Backward err in Gaussian elimination In lecture, I cut straight to the point: solving Ax b in finite precision using Gaussian elimination followed by fward and backward substitution yields a computed solution ˆx exactly satisfies (1 (A + δaˆx b, where δa 3nɛ mach ˆL Û, assuming ˆL and Û are the computed L and U facts. Though I didn t do this in class, here I will briefly sketch a part of the err analysis following Demmel s treatment ( Mostly, this is because I find the treatment in of our book less clear than I would like but also, the bound in Demmel s book is marginally tighter. Here is the idea. Suppose ˆL and Û are the computed L and U facts. We obtain û jk by repeatedly subtracting l ji u ik from the iginal a jk, i.e. ( û jk fl a jk ˆlji û ik. Regardless of the der of the sum, we get an err that looks like û jk a jk (1 + δ 0 ˆlji û ik (1 + δ i + O(ɛ 2 mach where δ i (j 1ɛ mach. Turning this around gives ( a jk 1 ˆljj û jk δ 0 ˆlji û ik (1 + δ i + O(ɛ 2 mach ˆl jj û jk (1 δ 0 + ˆlji û ik (1 + δ i δ 0 + O(ɛ 2 mach (ˆLÛ + E jk, jk where E jk ˆl jj û jk δ 0 + ˆlji û ik (δ i δ 0 + O(ɛ 2 mach

3 is bounded in magnitude by (j 1ɛ mach ( L U jk + O(ɛ 2 mach 1. argument f the components of ˆL yields A similar A ˆLÛ + E, where E nɛ mach ˆL Û + O(ɛ2 mach. In addition to the backward err due to the computation of the LU facts, there is also backward err in the fward and backward substitution phases, which gives the overall bound (1. Pivoting The backward err analysis in the previous section is not completely satisfacty, since L U may be much larger than A, yielding a large backward err overall. F example, consider the matrix A [ ] δ [ ] [ ] 1 0 δ 1 δ δ 1. If 0 < δ 1 then L U δ 2, even though A 2. The problem is that we ended up subtracting a huge multiple of the first row from the second row because δ is close to zero that is, the leading principle min is nearly singular. If δ were exactly zero, then the factization would fall apart even in exact arithmetic. The solution to the woes of singular and near singular mins is pivoting; instead of solving a system with A, we re-der the equations to get  [ ] 1 1 δ 1 [ ] [ ] δ δ Now the triangular facts f the re-dered system matrix  have very modest nms, and so we are happy. If we think of the re-dering as the effect of a permutation matrix P, we can write [ ] [ ] [ ] [ ] δ A P T LU δ δ 1 It s obvious that E j k is bounded in magnitude by 2(j 1ɛ mach ( L U jk + O(ɛ 2 mach. We cut a fact of two if we go down to the level of looking at the individual rounding errs during the dot product, because some of those errs cancel.

4 Note that this is equivalent to writing P A LU where P is another permutation (which undoes the action of P T. If we wish to control the multipliers, it s natural to choose the permutation P so that each of the multipliers is at most one. This standard choice leads to the following algithm: f j 1:n-1 end % Find ipiv > j to maximize A(i,j [absaij, ipiv] max(abs(a(j:n,j; ipiv ipiv + j-1; % Swap row ipiv and row j Aj A(j,j:n; A(j,j:n A(ipiv,j:n; A(ipiv,j:n Aj; % Recd the pivot row piv(j ipiv; % Update trailing submatrix A(j+1:n,j+1:n A(j+1:n,j+1:n - A(j+1:n,j*A(j,j+1:n; By design, this algithm produces an L fact in which all the elements are bounded by one. But what about the U fact? There exist pathological matrices f which the elements of U grow exponentially with n. But these examples are extremely uncommon in practice, and so, in general, Gaussian elimination with partial pivoting does indeed have a small backward err. Of course, the pivot growth is something that we can monit, so in the unlikely event that it does look like things are blowing up, we can tell there is a problem and try something different. When problems do occur, it is me frequently the result of ill-conditioning in the problem than of pivot growth during the factization.

5 Iterative refinement revisited Recall from last lecture that if we have a solver f  A + E with E small, then we can use iterative refinement to clean up the solution. The matrix  could come from finite precision Gaussian elimination of A, f example, from some factization of a nearby easier matrix. To get the refinement iteration, we take the equation (2 Ax Âx Ex b, and think of x as the fixed point f an iteration (3 Âx k+1 Ex k b. Note that this is the same as Âx k+1 ( Ax k b, x k+1 x k +  1 (b Ax k. Note that this latter fm is the same as inexact Newton iteration on the equation Ax k b 0 with the approximate Jacobian Â. If we subtract (2 from (3, we see Â(x k+1 x E(x k x 0, Taking nms, we have x k+1 x  1 E(x k x. x k+1 x  1 E x k x. Thus, if  1 E < 1, we are guaranteed that x k x as k. At least, this is what happens in exact arithmetic. In practice, the residual is usually computed with only finite precision, and so we might expect to stop making progress at some point. This is the topic of the first problem in HW 2.