Preconditioning. Noisy, Ill-Conditioned Linear Systems

Transcription

1 Preconditioning Noisy, Ill-Conditioned Linear Systems James G. Nagy Emory University Atlanta, GA Outline 1. The Basic Problem 2. Regularization / Iterative Methods 3. Preconditioning 4. Example: Image Restoration 5. Summary

2 Basic Problem Linear system of equations b = Ax where A, b are known A is large, structured Goal: Compute an approximation of x

3 Basic Problem Linear system of equations b = Ax + e where A, b are known A is large, structured, ill-conditioned Goal: Compute an approximation of x

4 Basic Problem Linear system of equations b = Ax + e where A, b are known A is large, structured, ill-conditioned Goal: Compute an approximation of x Applications: Ill-posed inverse problems. Geomagnetic Prospecting Tomography Image Restoration

5 Basic Problem Linear system of equations b = Ax + e where A, b are known A is large, structured, ill-conditioned Goal: Compute an approximation of x Applications: Ill-posed inverse problems. Geomagnetic Prospecting Tomography Image Restoration b = observed image A = blurring matrix (structured) e = noise x = true image

6 Basic Problem Linear system of equations b = Ax + e where A, b are known A is large, structured, ill-conditioned Goal: Compute an approximation of x Applications: Ill-posed inverse problems. Geomagnetic Prospecting Tomography Image Restoration b = observed image A = blurring matrix (structured) e = noise x = true image Example

7 Basic Problem Properties Computational difficulties revealed through SVD: Let A = UΣV T where Σ =diag(σ 1, σ 2,..., σ N ), σ 1 σ 2 σ N U T U = I, V T V = I

8 Basic Problem Properties Computational difficulties revealed through SVD: Let A = UΣV T where Σ =diag(σ 1, σ 2,..., σ N ), σ 1 σ 2 σ N U T U = I, V T V = I For ill-posed inverse problems, σ 1 1, small singular values cluster at small singular values oscillating singular vectors

9 Basic Problem Properties Inverse solution for noisy, ill-posed problems: If A = UΣV T, then x = A 1 b = V Σ 1 U T b = n i=1 u T i b σ i v i

10 Basic Problem Properties Inverse solution for noisy, ill-posed problems: If A = UΣV T, then ˆx = A 1 (b + e) = V Σ 1 U T (b + e) = n i=1 u T i (b + e) σ i v i

11 Basic Problem Properties Inverse solution for noisy, ill-posed problems: If A = UΣV T, then ˆx = A 1 (b + e) = V Σ 1 U T (b + e) = n i=1 u T i (b + e) σ i v i = n i=1 u T i b σ i v i + n i=1 u T i e σ i v i = x + error

12 Basic Problem Properties Inverse solution for noisy, ill-posed problems: If A = UΣV T, then ˆx = A 1 (b + e) = V Σ 1 U T (b + e) = n i=1 u T i (b + e) σ i v i = n i=1 u T i b σ i v i + n i=1 u T i e σ i v i = x + error Example

13 Regularization Basic Idea: Filter out effects of small singular values. x reg = n i=1 φ i u T i b σ i v i where the filter factors satisfy φ i { 1 if σi is large if σ i is small

14 Regularization Some regularization methods: 1. Truncated SVD x tsvd = k i=1 u T i b σ i v i 2. Tikhonov x tik = n i=1 σi 2 u T σi 2 i b + v i α2 σ i 3. Wiener x wien = n i=1 δ i σ 2 i δ i σ 2 i + γ2 i u T i b σ i v i

15 Iterative Regularization Basic idea: Use an iterative method (e.g., conjugate gradients) Terminate iteration before theoretical convergence: Early iterations reconstruct solution. Later iterations reconstruct noise.

16 Iterative Regularization Basic idea: Use an iterative method (e.g., conjugate gradients) Terminate iteration before theoretical convergence: Early iterations reconstruct solution. Later iterations reconstruct noise. Some important methods: CGLS, LSQR, GMRES MR2 (Hanke, 95) MRNSD (Kaufman, 93; N., Strakos, )

17 Iterative Regularization Efficient for large problems, provided 1. Multiplication with A is not expensive. 2. Convergence is rapid enough.

18 Iterative Regularization Efficient for large problems, provided 1. Multiplication with A is not expensive. Image restoration Use FFTs 2. Convergence is rapid enough. CGLS, LSQR, GMRES, MR2 often fast, especially for severely ill-posed, noisy problems. MRNSD based on steepest descent, typically converges very slowly.

19 Iterative Regularization Efficient for large problems, provided 1. Multiplication with A is not expensive. Image restoration Use FFTs 2. Convergence is rapid enough. CGLS, LSQR, GMRES, MR2 often fast, especially for severely ill-posed, noisy problems. MRNSD based on steepest descent, typically converges very slowly. Example

20 Preconditioning Purposes of preconditioning: 1. Accelerate convergence. Apply iterative method to P 1 Ax = P 1 b.

21 Preconditioning Purposes of preconditioning: 1. Accelerate convergence. Apply iterative method to P 1 Ax = P 1 b. In this case we minimize x Enforce regularization constraint on solution. (Hanke, 92; Hansen, 98) Apply iterative method to AL 1 Lx = b. In this case, we minimize Lx 2.

22 Preconditioning for Regularization Basic idea: Find a matrix L to enforce smoothness constraint min Lx 2 Typically L approximates a derivative operator.

23 Preconditioning for Speed Typical approach for Ax = b Find matrix P so that P 1 A I. Ideal choice: P = A In this case, converge in one iteration to x = A 1 b

24 Preconditioning for Speed Typical approach for Ax = b Find matrix P so that P 1 A I. Ideal choice: P = A In this case, converge in one iteration to x = A 1 b For ill-conditioned, noisy problems Inverse solution is corrupted with noise

25 Preconditioning for Speed Typical approach for Ax = b Find matrix P so that P 1 A I. Ideal choice: P = A In this case, converge in one iteration to x = A 1 b For ill-conditioned, noisy problems Inverse solution is corrupted with noise Ideal regularized preconditioner: If A = UΣV T (Hanke, N., Plemmons, 93) P = UΣ k V T = Udiag(σ 1,..., σ k,1,..., 1)V T

26 Preconditioning for Speed Notice that the preconditioned system is: P 1 A = (UΣ k V T ) 1 (UΣV T ) = V Σ 1 k ΣV T = V V T where =diag(1,..., 1, σ k+1,..., σ n ) That is, Large (good) singular values clustered at 1. Small (bad) singular values not clustered.

27 Preconditioning for Speed Remaining questions: 1. How to choose truncation index, k? Use regularization parameter choice methods, e.g., GCV, L-curve, Picard condition 2. We can t compute SVD, so now what? Use SVD approximation.

28 Preconditioning for Speed An SVD Approximation: Decompose A as: (Van Loan and Pitsianis, 93) A = C 1 D 1 + C 2 D C k D k where C 1 D 1 = argmin A C D F.

29 Preconditioning for Speed An SVD Approximation: Decompose A as: (Van Loan and Pitsianis, 93) A = C 1 D 1 + C 2 D C k D k where C 1 D 1 = argmin A C D F. Choose a structured (or sparse) U and V.

30 Preconditioning for Speed An SVD Approximation: Decompose A as: (Van Loan and Pitsianis, 93) A = C 1 D 1 + C 2 D C k D k where C 1 D 1 = argmin A C D F. Choose a structured (or sparse) U and V. Let Σ = argmin A UΣV T F.

31 Preconditioning for Speed An SVD Approximation: Decompose A as: (Van Loan and Pitsianis, 93) A = C 1 D 1 + C 2 D C k D k where C 1 D 1 = argmin A C D F. Choose a structured (or sparse) U and V. Let Σ = argmin A UΣV T F. That is, Σ = diag ( U T AV )

32 Preconditioning for Speed An SVD Approximation: Decompose A as: (Van Loan and Pitsianis, 93) A = C 1 D 1 + C 2 D C k D k where C 1 D 1 = argmin A C D F. Choose a structured (or sparse) U and V. Let Σ = argmin A UΣV T F. That is, Σ = diag ( U T AV ) = diag U T k i=1 C i D i V

33 Preconditioning for Speed Choices for U and V depend on problem (application). Since A = C 1 D 1 + C 2 D C k D k and C 1 D 1 = argmin A C D F we might use singular vectors of C 1 D 1.

34 Preconditioning for Speed Choices for U and V depend on problem (application). Since A = C 1 D 1 + C 2 D C k D k and C 1 D 1 = argmin A C D F we might use singular vectors of C 1 D 1. For image restoration, we also use Fourier Transforms (FFTs) Discrete Cosine Transforms (DCTs)

35 Example: Image Restoration 1. Matrix Structure 2. Efficiently computing SVD approximation

36 Matrix Structure in Image Restoration First, how do we get the matrix, A?

37 Matrix Structure in Image Restoration First, how do we get the matrix, A? Using linear algebra notation, the i-th column of A can be written as: Ae i = [ a 1 a i a n ]. 1. = a i

38 Matrix Structure in Image Restoration First, how do we get the matrix, A? Using linear algebra notation, the i-th column of A can be written as: Ae i = [ a 1 a i a n ] In an imaging system, e i = point source. 1. = a i Ae i = point spread function (PSF)

39 Matrix Structure in Image Restoration point source PSF

40 Matrix Structure in Image Restoration Spatially invariant PSF implies: e i e j e k Ae i Ae j Ae k

41 Matrix Structure in Image Restoration That is, spatially invariant implies Each column of A is identical, modulo shift. One point PSF is enough to fully describe A. A has Toeplitz structure.

42 Matrix Structure in Image Restoration e 5 = 1 1 blur p 11 p 12 p 13 p 21 p 22 p 23 p 31 p 32 p 33 Ae 5 = p 11 p 12 p 13 p 21 p 22 p 23 p 31 p 32 p 33 A = p 11 p 12 p 13 p 21 p 22 p 23 p 31 p 32 p 33

43 Matrix Structure in Image Restoration e 5 = 1 1 blur p 11 p 12 p 13 p 21 p 22 p 23 p 31 p 32 p 33 Ae 5 = p 11 p 12 p 13 p 21 p 22 p 23 p 31 p 32 p 33 A = p 12 p 11 p 13 p 12 p 11 p 13 p 12 p 22 p 21 p 23 p 22 p 21 p 23 p 22 p 32 p 31 p 33 p 32 p 31 p 33 p 32

44 Matrix Structure in Image Restoration e 5 = 1 1 blur p 11 p 12 p 13 p 21 p 22 p 23 p 31 p 32 p 33 Ae 5 = p 11 p 12 p 13 p 21 p 22 p 23 p 31 p 32 p 33 A = p 22 p 21 p 12 p 11 p 23 p 22 p 21 p 13 p 12 p 11 p 23 p 22 p 13 p 12 p 32 p 31 p 22 p 21 p 12 p 11 p 33 p 32 p 31 p 23 p 22 p 21 p 13 p 12 p 11 p 33 p 32 p 23 p 22 p 13 p 12 p 32 p 31 p 22 p 21 p 33 p 32 p 31 p 23 p 22 p 21 p 33 p 32 p 23 p 22

45 Matrix Structure in Image Restoration Matrix Summary Boundary Condition Matrix Structure zero BTTB periodic BCCB reflexive BTTB+BTHB+BHTB+BHHB B = block T = Toeplitz C = circulant H = Hankel

46 Matrix Structure in Image Restoration Matrix Summary Boundary Condition Matrix Structure zero BTTB periodic reflexive (strongly symmetric) BCCB (use FFT) BTTB+BTHB+BHTB+BHHB (use DCT) B = block T = Toeplitz C = circulant H = Hankel

47 Matrix Structure in Image Restoration For a separable PSF, we get: p 11 p 12 p 13 p 21 p 22 p 23 p 31 p 32 p 33 = cd T = c 1d 1 c 1 d 2 c 1 d 3 c 2 d 1 c 2 d 2 c 2 d 3 c 3 d 1 c 3 d 2 c 3 d 3 Ae 5 = c 1 c 2 c 3 d 1 d 2 d 3 d 1 d 2 d 3 d 1 d 2 d 3

48 Matrix Structure in Image Restoration For a separable PSF, we get: p 11 p 12 p 13 p 21 p 22 p 23 p 31 p 32 p 33 = cd T = c 1d 1 c 1 d 2 c 1 d 3 c 2 d 1 c 2 d 2 c 2 d 3 c 3 d 1 c 3 d 2 c 3 d 3 Ae 5 = c 1 c 2 c 3 d 1 d 2 d 3 d 1 d 2 d 3 d 1 d 2 d 3 c 1 c 2 c 3 d 1 d 2 d 3 d 1 d 2 d 3 d 1 d 2 d 3

49 Matrix Structure in Image Restoration For a separable PSF, we get: p 11 p 12 p 13 p 21 p 22 p 23 p 31 p 32 p 33 = cd T = c 1d 1 c 1 d 2 c 1 d 3 c 2 d 1 c 2 d 2 c 2 d 3 c 3 d 1 c 3 d 2 c 3 d 3 Ae 5 = c 1 c 2 c 3 d 1 d 2 d 3 d 1 d 2 d 3 d 1 d 2 d 3 c 1 c 2 c 3 d 2 d 1 d 3 d 2 d 1 d 3 d 2 d 2 d 1 d 3 d 2 d 1 d 3 d 2 d 2 d 1 d 3 d 2 d 1 d 3 d 2

50 Matrix Structure in Image Restoration For a separable PSF, we get: p 11 p 12 p 13 p 21 p 22 p 23 p 31 p 32 p 33 = cd T = c 1d 1 c 1 d 2 c 1 d 3 c 2 d 1 c 2 d 2 c 2 d 3 c 3 d 1 c 3 d 2 c 3 d 3 Ae 5 = c 1 c 2 c 3 d 1 d 2 d 3 d 1 d 2 d 3 d 1 d 2 d 3 c 2 c 3 d 2 d 1 d 3 d 2 d 1 d 3 d 2 d 2 d 1 d 3 d 2 d 1 d 3 d 2 c 1 d 2 d 1 d 3 d 2 d 1 d 3 d 2 c 2 d 2 d 1 d 3 d 2 d 1 d 3 d 2 c 3 d 2 d 1 d 3 d 2 d 1 d 3 d 2 c 1 d 2 d 1 d 3 d 2 d 1 d 3 d 2 c 2 d 2 d 1 d 3 d 2 d 1 d 3 d 2 = C D

51 Matrix Structure in Image Restoration If the PSF is not separable, we can still compute: P = r i=1 c i d T i (sum of rank-1 matrices) and therefore, get A = r i=1 C i D i (sum of Kron. products) In fact, we can get optimal decompositions. (Kamm, N, ; N., Ng, Perrone, 3)

52 SVD Approximation for Image Restoration Use A UΣV T, where If F ( C i D i ) F is best, U = V = F, Σ = diag ( F ( Ci D i ) F ) If C ( C i D i ) C T is best, U = V = C T, Σ = diag ( C ( Ci D i ) C T ) If (U c U d ) T ( C i D i ) (V c V d ) is best, U = U c U d, V = V c V d, Σ = diag ( (U c U d ) T ( Ci D i ) (Vc V d ) ) Example

53 The End Preconditioning ill-posed problems is difficult, possible. but Can build approximate SVD from Kronecker product approximations. Can implement efficiently for image restoration. Matlab software: RestoreTools (Lee, N., Perrone, 2) Object oriented approach for image restoration. nagy/restoretools/ Related software for ill-posed problems (Hansen, Jacobsen) pch/regutools/