A Hybrid LSQR Regularization Parameter Estimation Algorithm for Large Scale Problems

Transcription

1 A Hybrid LSQR Regularization Parameter Estimation Algorithm for Large Scale Problems Rosemary Renaut Joint work with Jodi Mead and Iveta Hnetynkova SIAM Annual Meeting July 10, 2009 National Science Foundation: Division of Computational Mathematics 1 / 24

2 Outline 1 The Problem 2 The Chi Squared Principle: Some Results 3 A Hybrid Method for Large Scale Problems 4 Conclusions National Science Foundation: Division of Computational Mathematics 2 / 24

3 Least Squares for Ax = b: A Quick Review Consider discrete systems: A R m n, b R m, x R n Ax = b + e, Classical Approach Linear Least Squares (A full rank) x LS = arg min x Ax b 2 2 Difficulty x LS sensitive to changes in right hand side b when A is ill-conditioned. For convolutional models system is ill-posed. National Science Foundation: Division of Computational Mathematics 3 / 24

4 Example Signal Restoration (a) Original (b) Noisy (c) Unregularized (d) Regularized National Science Foundation: Division of Computational Mathematics 4 / 24

5 The Weighting Matrix: Statistical Information on Measurements Some Assumptions for Multiple Data Measurements Given multiple measurements of data b: Usually error in b, e is an m vector of random measurement errors with mean 0 and positive definite covariance matrix C b = E(ee T ). For uncorrelated measurements C b is diagonal matrix of standard deviations of the errors. (Colored noise) For white noise C b = σ 2 I. Weighting by W b = C b 1 in data fit term, theoretically, ẽ = W b 1/2 e are uncorrelated. Difficulty if W b increases ill-conditioning of A! National Science Foundation: Division of Computational Mathematics 5 / 24

6 Formulation: Generalized Tikhonov Regularization With Weighting Regularized Functional ˆx = argmin J(x) = argmin{ Ax b 2 W b + λ 2 D(x x 0 ) 2 }. (1) D is a suitable operator, often derivative approximation. Assume N (A) N (D) = {0} x 0 is a reference solution, often x 0 = 0. Given D, W b goal is to find regularization parameter λ National Science Foundation: Division of Computational Mathematics 6 / 24

7 Choice of λ crucial Different algorithms yield different solutions. Discrepancy Principle L-Curve (many solutions) Generalized Cross Validation (GCV) (Calculate a minimum) Unbiased Predictive Risk (UPRE) (Calculate a trace) Examine statistical properties of the residual χ 2 Principle - extends discrepancy National Science Foundation: Division of Computational Mathematics 7 / 24

8 Statistics of the Regularized Least Squares Problem Thm: χ 2 distribution of the regularized functional (Renaut/Mead 2008) (NOTE: Weighting Matrix on Regularization term.) ˆx = argmin J D (x) = argmin{ Ax b 2 W b + (x x 0 ) 2 W D }, W D = D T W x D. (2) Assume Then W b and W x are symmetric positive definite. Problem is uniquely solvable N (A) N (D) = {0}. Moore-Penrose generalized inverse of W D is C D Statistics: Errors in the right hand side e N(0, C b ), and x 0 is known so that (x x 0 ) = f N(0, C D ), x 0 is the mean vector of the model parameters. J D (ˆx(W D )) χ 2 (m + p n) National Science Foundation: Division of Computational Mathematics 8 / 24

9 Significance of the χ 2 result For sufficiently large m = m + p n J D χ 2 (m + p n) E(J(x(W D ))) = m + p n E(JJ T ) = 2(m + p n) Moreover m 2 mz α/2 < J(ˆx(W D )) < m + 2 mz α/2. (3) z α/2 is the relevant z-value for a χ 2 -distribution with m = m + p n degrees National Science Foundation: Division of Computational Mathematics 9 / 24

10 When mean of the parameters is not known, or x 0 = 0 is not the mean Corollary: non-central χ 2 distribution of the regularized functional Recall ˆx = argmin J D (x) = argmin{ Ax b 2 W b + (x x 0 ) 2 W D }, W D = D T W x D. Assume all assumptions as before, but x x 0 is the mean vector of the model parameters. Let c = c 2 2 = QU T W b 1/2 A(x x 0 ) 2 2 Then J D χ 2 (m + p n, c) The functional at optimum follows a non central χ 2 distribution National Science Foundation: Division of Computational Mathematics 10 / 24

11 A further result when A is not of full column rank The Rank Deficient Solution Suppose A is not full column rank. Then the filtered solution can be written in terms of the GSVD x FILT (λ) = p i=p+1 r γ 2 i υ i (γ 2 i + λ 2 ) s i x i + n i=p+1 s i x i = p i=1 f i υ i s i x i + n i=p+1 s i x i. Here f i = 0, i = 1 : p r, f i = γ 2 i /(γ2 i + λ 2 ), i = p r + 1 : p. This yields J(x FILT (λ)) χ 2 (m n + r, c) n r components are not regularized notice degrees of freedom are reduced. National Science Foundation: Division of Computational Mathematics 11 / 24

12 General Result The Cost Functional follows a χ 2 Statistical Distribution Suppose degrees of freedom m and centrality parameter c then E(J D ) = m + c E(J D J T D) = 2( m) + 4c Suggests: Try to find W D so that E(J) = m + c Mead/Renaut introduced efficient Newton algorithm, c = 0 general W D. First find λ only. Find W x = λ 2 I National Science Foundation: Division of Computational Mathematics 12 / 24

13 Relation Discrepancy and χ 2 Principles Suppose noise is white: C b = σ 2 b I. W b its inverse. Discrepancy: λ such that residual satisfies m = b Ax(λ) 2 W b. (4) χ 2 : λ such that regularized residual satisfies m = b Ax(λ) 2 W b + λ 2 x x 0 2. (5) Both can be implemented by a Newton root finding algorithm. But discrepancy principle typically oversmooths. National Science Foundation: Division of Computational Mathematics 13 / 24

14 Relationship to Discrepancy Principle: GSVD Formulation Use GSVD of [W 1/2 b A, D]. Let λ = 1/σ x. For γ i the generalized singular values, and s = U T W 1/2 b r m = m n + p s i = s i /(γi 2σ2 x + 1), i = 1,..., p, t i = s i γ i. Discrepancy principle solves p 1 m ( γi 2σ2 x + 1 )2 s 2 i + s 2 i = m, (6) χ 2 principle solves i=1 F(σ x ) = p 1 ( γi 2σ2 x + 1 )s2 i + i=1 Equivalently: solve F = 0, where i=n+1 m i=n+1 s 2 i m = 0 F(σ x ) = s T s m and F (σ x ) = 2σ x t 2 2. National Science Foundation: Division of Computational Mathematics 14 / 24

15 The Newton Algorithm Without the GSVD Newton to Solve F(σ) = J D (σ) m = 0 We use σ = 1/λ, and y(σ (k) ) is the current solution for which x(σ (k) ) = y(σ (k) ) + x 0 Then σ J(σ) = 2 σ 3 Dy(σ) 2 < 0 Hence we have a basic Newton Iteration σ (k+1) = σ (k) ( ( σ(k) Dy )2 (J D (σ (k) ) m)). Need to estimate y and J efficiently National Science Foundation: Division of Computational Mathematics 15 / 24

16 LSQR Algorithm (Paige and Saunders - Golub-Kahan bidiagonalization) Only requires matrix-vector multipliciations with matrices A and A T. Initial vectors g 0 0, h 1 b/β 1. where β 1 b 0. Orthonormal vectors g i, h i, i = 1, 2,... are columns of matrices H j [ h 1,..., h j ] R m j, G j [ g 1,..., g j ] R n j. Recurrences can be written in the matrix notation L j A T H j = G j Lj T, A G j = [ H j, h j+1 ] L j+. (7) α 1 β 2 α β j α j, [ L L j j+ β j+1 e T j The j steps of the bidiagonalization yield a subproblem L j+ y j e 1 β 1 and x j (σ) G j y j (σ). ], L j+ R (j+1) j. National Science Foundation: Division of Computational Mathematics 16 / 24

17 Regularizing the subproblem With regularization when D = I, y j (σ) solves regularized problem L j+ y j e 1 β /σ 2 y j 2 Note that x j (σ) 2 2 = G jy j (σ) 2 = y j (σ) 2 2. The regularized problem is of size j + 1 j. j m Standard Methods can be applied to find σ efficiently. Nagy uses a weighted GCV We propose the χ 2 principle. Note that we apply directly to subproblem. Degrees of freedom are j = j + 1 National Science Foundation: Division of Computational Mathematics 17 / 24

18 Alternative Implementing on the Global problem The Hybrid projects to a subproblem and then regularizes A modified hybrid does the Newton iteration during the bidiagonalization For each subproblem created through the iteration, σ is updated. One step of Newton is taken each subproblem The degrees of freedom are determined by the original problem. m - strictly that is not correct. National Science Foundation: Division of Computational Mathematics 18 / 24

19 Illustrating Deblurring Result (Global hybrid): Problem Size Example taken from RESTORE TOOLS Nagy et al : 15% Noise Computational Cost is Minimal: Projected Problem Size is 15, λ =.58 National Science Foundation: Division of Computational Mathematics 19 / 24

20 Problem Grain noise 15% added for increasing subproblem size Signal to noise ratio 10 log 10 (1/e) relative error e Regularization parameter σ in each case. National Science Foundation: Division of Computational Mathematics 20 / 24

21 Illustrating the progress of the Newton algorithm with LSQR National Science Foundation: Division of Computational Mathematics 21 / 24

22 Illustrating the progress of the Newton algorithm Hybrid LSQR National Science Foundation: Division of Computational Mathematics 22 / 24

23 Signal to noise ratio 10 log 10 (1/e) relative error e National Science Foundation: Division of Computational Mathematics 23 / 24 Problem Grain noise 15% added for increasing subproblem size

24 Conclusions Observations χ 2 principle can be used for large scale problems. Preconditioning can be applied to weighted problem. PROBLEM SIZE of Hybrid LSQR USED IS VERY SMALL Method is very efficient, Newton method is robust and fast. a priori information is needed. Can be obtained directly from the data. e.g. Use local statistical information of image. National Science Foundation: Division of Computational Mathematics 24 / 24