Lanczos Bidiagonalization with Subspace Augmentation for Discrete Inverse Problems

Transcription

1 Lanczos Bidiagonalization with Subspace Augmentation for Discrete Inverse Problems Per Christian Hansen Technical University of Denmark Ongoing work with Kuniyoshi Abe, Gifu Dedicated to Dianne P. O Leary

2 Many Many Thanks to Dianne for Inspiration, Insight, and Very Nice Collaborations Dianne, Jim and I wrote a book and tried it on students in Bari. It is water! Pictures by Nicola Mastronardi 2/20 P. C. Hansen Lanczos Bidiagonalization with Subspace Augmentation (for Dianne)

3 Overview of Talk Discrete inverse problem: Ax = b 1. Iterative Krylov-subspace methods regularizing iterations. 2. Augmenting the Krylov subspace for improved solutions. 3. Lanczos bidiagonalization algorithm with augmented subspace. 4. Numerical examples that illustrate the advantage of this idea. 3/20 P. C. Hansen Lanczos Bidiagonalization with Subspace Augmentation (for Dianne)

4 Regularization Algorithms Variational formulations take the form min ka x bk R(x) ª x where R(x) is a regularization terms that penalizes unwanted features in the solution, and is a user-chosen regularization parameter. H & O Leary 1993, O Leary 2001; Rust & O Leary 2008 choosing λ. Projection formulations take the form min x ka x bk 2 2 s.t. x 2 S k ; where the \signal subspace" S k is a linear subspace of dimension k. If S k is chosen such that it captures the main features in the solution, then this approach is well suited for large-scale problems. Hybrid methods that apply regularization to the projected problem. Chung, Nagy & O Leary 2008 hybrid method with GCV. 4/20 P. C. Hansen Lanczos Bidiagonalization with Subspace Augmentation (for Dianne)

5 The Signal Subspace In some applications we can use a pre-determined subspace, e.g., spanned by the Fourier basis, the discrete cosine bases, a wavelet basis, etc. An example: truncated SVD S k = spanfv 1 ; v 2 ; : : : ; v k g: Alternatively we can use a subspace determined by the given problem, e.g., the Krylov subspace K k associated with a speci c iterative method CGLS : spanfa T b; A T A A T b; (A T A) 2 A T b; : : :g ; GMRES : spanfb; A b; A 2 b; : : :g ; RRGMRES : spanfa b; A 2 b; A 3 b; : : :g : O Leary & Simmons 1981, Kilmer & O Leary 2001 regularizing iterations. 5/20 P. C. Hansen Lanczos Bidiagonalization with Subspace Augmentation (for Dianne)

6 Illustration of Semi-Convergence A y b ¹x = exact solution 6/20 P. C. Hansen Lanczos Bidiagonalization with Subspace Augmentation (for Dianne)

7 Augmented Signal Subspace Let W p denote a linear subspace that captures additional speci c components of the desired solution; dim(w p ) = p k = no. its. Then it can be advantageous to use an augmented linear subspace S p;k = W p + K k ; W p = R(W p ) = spanfw 1 ; : : : ; w p g : Ex.: deriv2 & GMRES. All vectors in the Krylov subspace! 0 at end points. Now use w 1 = (1; 1; : : : ; 1) T, w 2 = (1; 2; : : : ; n) T. GMRES Augmented GMRES Here we want an e±cient CGLS-type algorithm to solve the problem min x ka x bk 2 2 s.t. x 2 S p;k = W p + K k (A T A; A T b) : 7/20 P. C. Hansen Lanczos Bidiagonalization with Subspace Augmentation (for Dianne)

8 Overview of Methods Square matrix A 2 R n n ² \Augmented (RR)GMRES" (Baglama, Reichel 2007), where the subspace augmentation idea was originally formulated. An elegant and e±cient algorithmthat uses an incorrect subspace. ² \R 3 GMRES" (Dong, Garde, H 2014), uses the correct subspace, less elegant, still e±cient. Rectangular matrix A 2 R m n (this work) ² In some problems (e.g., tomography) the matrix A is rectangular. ² In some problems (tomography, inverse heat equation) the Arnoldi subspace is not suited. ² \LBAS" { Lanczos bidiagonalization with augmented subspace. ² Open question: can we use LSQR or LSMR to implement this? 8/20 P. C. Hansen Lanczos Bidiagonalization with Subspace Augmentation (for Dianne)

9 Towards our Algorithm LBAS We want to solve min x ka x bk 2 2 s.t. x 2 W p + K k (A T A; A T b) : In principle we could use, say, a Hessenberg decomposition A [ W p ; A T b ; A T A A T b ; ; (A T A) k 1 A T b ] = V p+k+1 H p+k and compute the solution as x (k) = [ W p ; A T b ; A T A A T b ; : : : ; (A T A) k 1 A T b ] y (k) ; y (k) = argmin y kh p+1 y V T p+k+1bk 2 2 : But we prefer to use a stable and e±cient \standard" algorithm. Run the bidiagonalization algorithm to compute an orthonormal basis of K k (A T A; A T b), and augment it by W p in each step of the algorithm. This seems cumbersome { but the overhead is favorably small! 9/20 P. C. Hansen Lanczos Bidiagonalization with Subspace Augmentation (for Dianne)

10 Setting the Stage for Our Algorithm At step k we have the decomposition h A [ V k ; W p ] = U k+1 ; U e i B k G k k 0 F k where ² A V k = U k+1 B k is obtained after k steps of the bidiag. process. ² V k 2 R n k has orthonormal columns that span K j (A T A; A T b). ² U k+1 2 R m (k+1) has orthonormal columns, u 1 = b=kbk 2. ² eu k 2 R m p : range(aw p ) = range(u k+1 G k + eu k F k ) and eu k T U k+1 = 0. ² B k 2 R (k+1) k is a lower bidiagonal matrix. ² F k 2 R p p and changes in every iteration. ² G k is (k + 1) p and is updated along with B k. The columns of [ V j ; W p ] form a basis for S p;j. 10/20 P. C. Hansen Lanczos Bidiagonalization with Subspace Augmentation (for Dianne)

11 More Details Recall that A [ V k ; W p ] = h U k+1 ; U e i B k G k k 0 F k : The matrices G k 2 R (k+1) p and F k 2 R p p are composed of the coe±cients of AW p with respect to basis of range(u k+1 ) and range( e U k ), respectively: G k = U T k+1aw p ; F k = e U T k AW p : Then the iterate x (k) 2 S p;k is given by x (k) = [ V k ; W p ] y (k), where " # " # y (k) Bk G k U T 2 k+1 = argmin y y b : 0 F k eu T j 2 11/20 P. C. Hansen Lanczos Bidiagonalization with Subspace Augmentation (for Dianne)

12 Algorithm: LBAS 1. Set U 1 = b=kbk 2, V 0 = [ ], B 0 = [ ], G 0 = U T 1 AW p, and k = Use the bidiag. process to obtain v k, u k+1 such that A V k = U k+1 B k, where 2 3 V k = [V k 1 ; v k ], U k+1 = [ U k ; u k+1 ], B k = 4 B 0 k G 3. Compute G k = k 1 u T k+1 AW 2 R (k+1) p. p 4. Orthonormalize AW p with respect to U k+1 to obtain e U k 2 R m p. 5. Compute F k = U e k T AW p 2 R p p. " # " Bk G k U T k+1 6. Solve min y y 0 F k 7. Then x (k) = [ V k ; W p ] y (k). eu T k # b 8. Stop, or set k := k + 1 and return to step to obtain y (k). Recomputation of e U k and F k in each step; but p is small! 12/20 P. C. Hansen Lanczos Bidiagonalization with Subspace Augmentation (for Dianne)

13 Efficient and Stable Implementation In each step we update the orthogonal factorization: 2 " # T (11) k T (12) 3 k Bk G k 6 = Q 4 0 T (22) 7 5 ; 0 F k k 0 0 T (11) k 2 R k k and T (22) k 2 R p p are upper triangular, Q is orthogonal. Update T (11) k via Givens rotations that are also applied to G k and U T k+1 b. eu k is already orthogonal to U k, hence (in principle) we can perform the update eu k+1 = (I m u k+1 u T k+1 ) e U k : For numerical stability: must reorthogonalize the columns of V k, U k+1, and e U k. Consider the use of partial reorthogonalization. Algorithm HYBR (Chung, Nagy, O Leary 2008) uses full reorthogonalization. 13/20 P. C. Hansen Lanczos Bidiagonalization with Subspace Augmentation (for Dianne)

14 Numerical Examples Setting up the test problems: 1. Generate noise-free system: A x exact = b exact. 2. Add noise: b = b exact + e where e is a random vector of Gaussian white noise scaled such that kek 2 =kb exact k 2 =. 3. We show best solution within the iterations plus: ² relative error kx exact x (k) k 2 =kx exact k 2, ² relative residual norm kb A x (k) k 2 =kbk 2. We compare combinations of the following algorithms: ² CGLS is the implementation from Regularization Tools. ² RRGMRES is the implementation from Regularization Tools. ² R 3 GMRES is our implementation (Dong, Garde, H 2014). ² LBAS is our new algorithm. 14/20 P. C. Hansen Lanczos Bidiagonalization with Subspace Augmentation (for Dianne)

15 Large Component in Augment. Subspace Test problem deriv2(n,2), n = 32, relative noise level = W 2 = spanfw 1 ; w 2 g; w 1 = (1; 1; : : : ; 1) T ; w 2 = (1; 2; : : : ; n) T : For this problem kw 2 W2 T x exact k 2 =kx exact k 2 = 0:99 ; k(i W 2 W2 T )x exactk 2 =kx exact k 2 = 0:035 ; we only need to spend e ort in capturing the small component in W 2?. 15/20 P. C. Hansen Lanczos Bidiagonalization with Subspace Augmentation (for Dianne)

16 Capture a Discontinuity Test problem gravity(n), n = 100, = 10 3, exact sol. changed to include a discontinuity between elements ` = 50 and ` + 1 = 51. Augmentation matrix W 2 allows us to represent this discontinuity: ones(`; 1) zeros(`; 1) w 1 = ; w zeros(n `; 1) 2 = : ones(n `; 1) 16/20 P. C. Hansen Lanczos Bidiagonalization with Subspace Augmentation (for Dianne)

17 Fix Boundary Conditions Here W 2 compensates for the \incorrect" or \incompatible" boundary conditions implicit in A, by allowing the regularized solutions to have nonzero values and nonzero derivatives at the endpoints. 17/20 P. C. Hansen Lanczos Bidiagonalization with Subspace Augmentation (for Dianne)

18 Fix Boundary Conditions, Rectangular A The matrix A is rectangular so RRGMRES and R 3 GMRES cannot be used. 18/20 P. C. Hansen Lanczos Bidiagonalization with Subspace Augmentation (for Dianne)

19 Compute Spectrum of X-Ray Source The spectrum of an X-ray source (where accelerated electrons hit an anode) consists of a continuous spectrum superimposed with line spectra. We know the frequencies of the line spectral, so we can easily incorporate this information through the augmentation subspace. Experiment with two choices: W delta two delta functions at the right frequencies, W Gauss two narrow Gauss functions at the right frequencies. Exact spectrum Many thanks to Jan Sijbers for inspiration to this example. 19/20 P. C. Hansen Lanczos Bidiagonalization with Subspace Augmentation (for Dianne)

20 Conclusions We consider (again) how to augment the Krylov subspace. Focus here on rectangular matrices and Lanczos bidiag. We develop an efficent algorihtm LBAS. Numerical examples demonstrate the advantage of LBAS. Future work: Selective reorthogonalization? Is it occasionally necessary to do the MGS twice? A similar algorithm based on MINRES/MR-II? Hybrid algorithm with regularization of projected problem! 20/20 P. C. Hansen Lanczos Bidiagonalization with Subspace Augmentation (for Dianne)