The Chi-squared Distribution of the Regularized Least Squares Functional for Regularization Parameter Estimation

Transcription

1 The Chi-squared Distribution of the Regularized Least Squares Functional for Regularization Parameter Estimation Rosemary Renaut DEPARTMENT OF MATHEMATICS AND STATISTICS Prague 2008 MATHEMATICS AND STATISTICS 1 / 40

2 Outline Introduction and Motivation Some Standard (or NOT) Methods for Regularization Parameter Estimation Statistical Results for Least Squares Implications of Statistical Results for Regularized Least Squares Newton algorithm Algorithm with LSQR Results Conclusions and Future Work Other Detailed Results Results for Validating LSQR Implementation with GSVD MATHEMATICS AND STATISTICS 2 / 40

3 Least Squares for Ax = b Consider discrete systems: A R m n, b R m, x R n Ax = b + e, Classical Approach Linear Least Squares x LS = arg min x Ax b 2 2 Difficulty x LS is sensitive to changes in the right hand side b when A is ill-conditioned. System is numerically ill-posed. MATHEMATICS AND STATISTICS 3 / 40

4 Handling the ill-posedness: Given multiple measurements of data Include some additional information about the solution and/or the data Usually error in b, e is an m vector of random measurement errors with mean 0 and positive definite covariance matrix C b = E(ee T ). Suppose C b is known. (Calculate if given multiple b) For uncorrelated measurements C b is diagonal matrix of standard deviations of the errors. (Colored noise) Perhaps the fit to data can be calculated in a weighted norm. Let W b = C b 1 and L b L b T = W b be the Choleski factorization of W b and weight the equation: L b Ax = L b b + ẽ, ie x WLS = arg min{ b Ax 2 x W b Then, theoretically, ẽ are uncorrelated. (White noise). But system ill-conditioning is usually deteriorated if noise is far from white (see Hansen). MATHEMATICS AND STATISTICS 4 / 40

5 Alternative: Introduce a Mechanism for Regularization Weighted Fidelity with Regularization Regularize x LS = arg min{ b Ax 2 x W b + λ 2 R(x)}, where R(x) is a regularization term λ is a regularization parameter which is unknown. Notice that the solution is x LS (λ), dependent on λ. It also depends on choice of R. Requirements Depends on R - what to chose? Depends on λ - what to chose? MATHEMATICS AND STATISTICS 5 / 40

6 A Specific Choice R(x) = D(x x 0 ) 2 : Tikhonov Regularized Generalized Tikhonov regularization: Given matrix D that is suitable. ˆx = argmin J(x) = argmin{ Ax b 2 W b + λ 2 D(x x 0 ) 2 }. (1) Assume N (A) N (D) = Weighting matrix W b is inverse covariance matrix for data b. x 0 is a reference solution, often x 0 = 0. Solution ˆx(λ) = argmin J(x) = argmin{ Ax b 2 W b + λ 2 D(x x 0 ) 2 }. (2) Question Given D, how do we find λ? MATHEMATICS AND STATISTICS 6 / 40

7 Some standard approaches I: L-curve - Find the corner Let r(λ) = (A(λ) A)b: Influence Matrix A(λ) = A(A T W b A+λ 2 D T D) 1 A T Plot log( Dx ), log( r(λ) ) Trade off contributions. Expensive - requires range of λ. GSVD makes calculations efficient. Not statistically based Find corner No corner MATHEMATICS AND STATISTICS 7 / 40

8 Some standard approaches II: Generalized Cross-Validation (GCV) Minimizes GCV function b Ax(λ) 2 W b [trace(i m A(λ))] 2, which estimates predictive risk. Expensive - requires range of λ. GSVD makes calculations efficient. Statistically based Requires minimum Multiple minima Sometimes flat MATHEMATICS AND STATISTICS 8 / 40

9 Some standard approaches III: Unbiased Predictive Risk Estimation (UPRE) Minimize expected value of predictive risk: Minimize UPRE function b Ax(λ) 2 W b +2 trace(a(λ)) m Expensive - requires range of λ. GSVD makes calculations efficient. Statistically based Minimum needed MATHEMATICS AND STATISTICS 9 / 40

10 Hybrid LSQR Methods to Project out the Noise Iterate LSQR to find solution and stop when noise starts to dominate (Hnetynkova and others) Solve the reduced system. Hybrid method - solve reduced system with additional regularization. Cost of regularization of reduced system is minimal Any regularization method may be used ( Nagy) Talk to the local experts! MATHEMATICS AND STATISTICS 10 / 40

11 A More general formulation: Maximum A Posteriori Method Formulation: ˆx = argmin J(x) = argmin{ Ax b 2 W b + (x x 0 ) 2 W D }. (3) Notice the regularization term includes a new weighting matrix, but no mapping D. (It is hidden in W D ) Standard: W D = λ 2 D T D, λ unknown penalty parameter. Ideally, statistically, W D is inverse covariance matrix for the mapped model Dx i.e. λ = 1/σ x, σ 2 x the common variance in Dx. Assumes the resulting estimates for Dx uncorrelated. ˆx is the standard maximum a posteriori (MAP) estimate of the solution, when all a priori information is provided. Can this provide additional information? MATHEMATICS AND STATISTICS 11 / 40

12 Background: Statistics of the Least Squares Problem Theorem (Rao73: First Fundamental Theorem) Let r be the rank of A and for b N(Ax, σb 2 I), (errors in measurements are normally distributed with mean 0 and covariance σb 2 I), then J = min Ax b 2 σ 2 x bχ 2 (m r). J follows a χ 2 distribution with m r degrees of freedom: Basically the Discrepancy Principle Corollary (Weighted Least Squares) For b N(Ax, C b ), and W b = C b 1 then J = min Ax b 2 x W b χ 2 (m r). MATHEMATICS AND STATISTICS 12 / 40

13 Extension: Statistics of the Regularized Least Squares Problem Two New Results to Help Find the Regularization parameter: Theorem: χ 2 distribution of the regularized functional ˆx = argmin J D (x) = argmin{ Ax b 2 W b + (x x 0 ) 2 W D }, W D = D T W x D. (4) Assume W b and W x are symmetric positive definite. Problem is uniquely solvable N (A) N (D) 0. Moore-Penrose generalized inverse of W D is C D Statistics: (b Ax) = e N(0, C b ), (x x 0 ) = f N(0, C D ), x0 is the mean vector of the model parameters. Then J D χ 2 (m + p n) MATHEMATICS AND STATISTICS 13 / 40

14 Corollary: a-priori information not mean value, e.g. x 0 = 0 Corollary: non-central χ 2 distribution of the regularized functional ˆx = argmin J D (x) = argmin{ Ax b 2 W b + (x x 0 ) 2 W D }, W D = D T W x D. (5) Assume all assumptions as before, but x 1 x 0 is the mean vector of the model parameters. Let c = c 2 2 = QU T W b 1/2 A(x 1 x 0 ) 2 2 Then J D χ 2 (m + p n, c) MATHEMATICS AND STATISTICS 14 / 40

15 Implications of the Result Statistical Distribution of the Functional Mean and Variance are prescribed E(J D ) = m + p n + c E(J D J T D) = 2(m + p n) + 4c Can we use this? YES Try to find W D so that E(J) = m n + p + 2c (Mead 2007) Mead algorithm finds W D but is expensive Our proposal - find λ only. MATHEMATICS AND STATISTICS 15 / 40

16 What do we need to apply the Theory? Requirements Covariance information C b on data parameters b ( or on model parameters x!) A priori information either x 0 is the mean, or mean value x 1. But x 1 and x 0 are not known. For repeated data measurements C b can be calculated. Also b 1 can be found, the mean of b. But E(b) = AE(x) implies b 1 = Ax 1. Hence c = c 2 2 = QU T W b 1/2 (b 1 Ax 0 ) 2 2 E(J D ) = E( QU T W b 1/2 (b Ax 0 ) 2 2) = m+p n+ QU T W b 1/2 (b 1 Ax 0 ) 2 2 Then we can use E(J) to find λ MATHEMATICS AND STATISTICS 16 / 40

17 Assume x 0 is the mean (experimentalists do know something about the model parameters) DESIGNING THE ALGORITHM: I Recall: if C b and C x are good estimates of covariance should be small. J D (ˆx) (m + p n) Thus, let m = m + p n then we want m 2 mz α/2 < J(x(W D )) < m + 2 mz α/2. (6) z α/2 is the relevant z-value for a χ 2 -distribution with m degrees GOAL Find W D to make (6) tight: Single Variable case find λ J D (ˆx(λ)) m MATHEMATICS AND STATISTICS 17 / 40

18 A Newton-line search Algorithm to find λ.(basic algebra) Newton to Solve F(σ) = J D (σ) m = 0 We use σ = 1/λ, and y(σ (k) ) is the current solution for which Then x(σ (k) ) = y(σ (k) ) + x 0 σ J(σ) = 2 σ 3 Dy(σ) 2 < 0 Hence we have a basic Newton Iteration Add a line search σ (k+1) = σ (k) ( ( σ(k) Dy )2 (J D (σ (k) ) m)). σ (k+1) = σ (k) (1 + α(k) 2 ( σ(k) Dy )2 (J D (σ (k) ) m)). MATHEMATICS AND STATISTICS 18 / 40

19 Algorithm Using the GSVD GSVD Use GSVD of [W 1/2 b A, D] For γ i the generalized singular values, and s = U T W 1/2 b r m = m n + p s i = s i /(γi 2σ2 x + 1), i = 1,..., p, t i = s i γ i. Find root of F(σ x ) = p 1 ( γi 2σ2 x + 1 )s2 i + i=1 m s 2 i m = 0 i=n+1 Equivalently: solve F = 0, where F(σ x ) = s T s m and F (σ x ) = 2σ x t 2 2. MATHEMATICS AND STATISTICS 19 / 40

20 Discussion on Convergence F is monotonic decreasing (F (σ x ) = 2σ x t 2 2 ) Solution either exists and is unique for positive σ Or no solution exists F(0) < 0. implies incorrect statistics of the model Theoretically, lim σ F > 0 possible. Equivalent to λ = 0. No regularization needed. MATHEMATICS AND STATISTICS 20 / 40

21 Practical Details of Algorithm Find the parameter Step 1: Bracket the root by logarithmic search on σ to handle the asymptotes: yields sigmamax and sigmamin Step 2: Calculate step, with steepness controlled by told. Let t = Dy/σ (k), where y is the current update, given from the GSVD, then step = 1 2 ( 1 max { t, told} )2 (J D (σ (k) ) m) Step 3: Introduce line search α (k) in Newton sigmanew = σ (k) (1 + α (k) step) α (k) chosen such that sigmanew within bracket. MATHEMATICS AND STATISTICS 21 / 40

22 Practical Details of Algorithm: Large Scale problems Algorithm Initialization Convert generalized Tikhonov problem to standard form. Use LSQR algorithm to find the bidiagonal matrix spanning appropriate space of solution Obtain a solution of the bidiagonal problem for given σ. Resuse bidiagonalization in update of σ for Newton. Each σ calculation of algorithm reuses saved information from the Lancos bidiagonalization. The system is augmented if needed. MATHEMATICS AND STATISTICS 22 / 40

23 Comparison with Standard LSQR hybrid Algorithm Algorithm concurrently regularizes and solves the system. In contrast, standard hybrid LSQR solves projected system with regularization. Needs only cost of standard LSQR algorithm with some updates for solution solves for iterated σ. The regularization introduced by LSQR projection may be useful for preventing problems with GSVD expansion. Makes algorithm viable for large scale problems. MATHEMATICS AND STATISTICS 23 / 40

24 Recall: Implementation Assumptions Covariance of Error: Statistics of Measurement Errors Information on the covariance structure of errors in b needed. Use C b = σb 2 I for common covariance, white noise. Use C b = diag(σ1 2, σ2 2,..., σ2 m) for colored uncorrelated noise. With no noise information C b = I. Use b 1 as the mean of measured b, when implemented with centrality parameter, x 0 = 0. Tolerance on Convergence The convergence tolerance depends on the noise structure. Use TOL = 2 mz α/2. No noise structure use α =.001, generates large TOL Good noise information use α =.95, generates small TOL MATHEMATICS AND STATISTICS 24 / 40

25 Recall: Implementation Assumptions Covariance of Error: Statistics of Measurement Errors Information on the covariance structure of errors in b needed. Use C b = σb 2 I for common covariance, white noise. Use C b = diag(σ1 2, σ2 2,..., σ2 m) for colored uncorrelated noise. With no noise information C b = I. Use b 1 as the mean of measured b, when implemented with centrality parameter, x 0 = 0. Tolerance on Convergence The convergence tolerance depends on the noise structure. Use TOL = 2 mz α/2. No noise structure use α =.001, generates large TOL Good noise information use α =.95, generates small TOL MATHEMATICS AND STATISTICS 24 / 40

26 An example of the method: Seismic Signal Restoration The Data Set and Goal Real data set of 48 signals of length The point spread function is derived from the signals. Calculate the signal variance pointwise over all 48 signals. Goal: restore the signal x from Ax = b, where A is psf matrix and b is given blurred signal. Method of Comparison- no exact solution known No exact solution. Downsample the signal and restore for different resolutions Resolution 2 : 1 5 : 1 10 : 1 20 : : 1 Points Do results converge? Compare with UPRE and L-Curve. MATHEMATICS AND STATISTICS 25 / 40

27 An example of the method: Seismic Signal Restoration The Data Set and Goal Real data set of 48 signals of length The point spread function is derived from the signals. Calculate the signal variance pointwise over all 48 signals. Goal: restore the signal x from Ax = b, where A is psf matrix and b is given blurred signal. Method of Comparison- no exact solution known No exact solution. Downsample the signal and restore for different resolutions Resolution 2 : 1 5 : 1 10 : 1 20 : : 1 Points Do results converge? Compare with UPRE and L-Curve. MATHEMATICS AND STATISTICS 25 / 40

28 Comparison High Resolution White noise (left) and Colored Noise (right) Greater contrast with χ 2. UPRE is insufficiently regularized. L-curve severely undersmooths (not shown). Parameters not consistent across resolutions MATHEMATICS AND STATISTICS 26 / 40

29 THE UPRE SOLUTION: White Noise and Colored Noise x 0 = 0 Regularization Parameters are consistent: σ = all resolutions MATHEMATICS AND STATISTICS 27 / 40

30 THE GSVD SOLUTION: White Noise (left) and Colored Noise (right) x 0 = 0 Regularization Parameters are consistent: σ = (left), σ = (right) all resolutions MATHEMATICS AND STATISTICS 28 / 40

31 THE NONCENTRAL GSVD SOLUTION: White Noise (left) and Colored Noise (right) x 0 = 0 Regularization quite consistent resolution 2 to 100 σ = , , , , (left) σ = ,.00007,.00007,.00007, (right). Notice that colored noise eliminates second arrival of signal but excellent contrast to identify primary arrival. MATHEMATICS AND STATISTICS 29 / 40

32 More Signals! UPRE and L-curve exhibit under regularization MATHEMATICS AND STATISTICS 30 / 40

33 Conclusions For the Case when x 0 known Observations A new statistical method for estimating regularization parameter Compares favorably with UPRE with respect to performance and compared to L-curve. (GCV is not competitive). Method can be used for large scale problems. Method is very efficient, Newton method is robust and fast. But x 0 is the mean of x is needed. MATHEMATICS AND STATISTICS 31 / 40

34 Difficulties when central parameter is required What are the issues? Function need not be monotonic More problematic for NonCentral version with x 0 not the mean. (ie x 0 = 0. σ can be bounded by result of central case. Range of σ given by range of γ i. May oversmooth the solution if good range of σ not found. MATHEMATICS AND STATISTICS 32 / 40

35 Future Work Other Results and Future Work Degrees of freedom reduced when using the GSVD. How to apply Picard condition for GSVD to handle problems with robustness due to conditioning of C b Image deblurring. (Implementation to use minimal storage) Diagonal Weighting Schemes Edge preserving regularization Constraint implementation ( with Mead submitted). MATHEMATICS AND STATISTICS 33 / 40

36 Some Small Scale Experiments: Verify Robustness of LSQR/GSVD Details Take example from Hansen s toolbox, eg shaw, phillips, heat, ilaplace. Generate 500 copies for each noise level, here.005,.01,.05,.1. Solve for 500 cases using GSVD and LSQR Newton. Pairwise t test on obtained σ: verify equivalence GSVD and LSQR. Compare results with statistical technique : UPRE Errors - relative least squares, and max error. Calculate over all errors less than.5. Regularization parameter (not given). MATHEMATICS AND STATISTICS 34 / 40

37 Example of the Colored Noise distribution The pointwise variance for each noise level MATHEMATICS AND STATISTICS 35 / 40

38 Noise and Exact Right Hand Side Vector Actual Noise Levels MATHEMATICS AND STATISTICS 36 / 40

39 Noise and Exact Right Hand Side Vector Actual Noise Levels MATHEMATICS AND STATISTICS 36 / 40

40 Noise and Exact Right Hand Side Vector Actual Noise Levels MATHEMATICS AND STATISTICS 36 / 40

41 Noise and Exact Right Hand Side Vector Actual Noise Levels MATHEMATICS AND STATISTICS 36 / 40

42 Problem size 64, Regularization First Order Derivative, Colored Noise shaw Bidiagonal Newton Steps P value ɛ Average GSVD LSQR Iteration σ 5e e e e e e 01 ilaplace 5e e e 01 1e e e 01 heat 5e e e e e e + 00 phillips 5e e e e e e + 00 Table: Convergence characteristics and P-values comparing GSVD and LSQR iteration steps and σ values MATHEMATICS AND STATISTICS 37 / 40

43 Problem size 64, Regularization First Order Derivative, Colored Noise Least Squares Error shaw Max Error ɛ GSVD LSQR UPRE GSVD LSQR UPRE.05.34(207).34(207).34(146).31(111).31(111).31(84).1.39(105).39(105).35(29).35(43).35(43).35(28) ilaplace.05.23(361).23(361).16(398).12(495).12(495).07(425).1.25(317).25(317).21(370).15(490).15(490).10(428) heat.05.27(500).27(500).28(479).19(500).19(500).20(489).1.35(497).35(497).36(461).27(500).27(500).27(489) phillips.05.10(500).10(500).10(455).10(500).10(500).09(455).1.12(500).12(500).11(431).12(500).12(500).11(431) Table: Comparison with UPRE, Relative Least Squares and max error, in parentheses the number of accepted values, large values (>.5) excluded from the average MATHEMATICS AND STATISTICS 38 / 40

44 Summary of Results Major Observations GSVD-LSQR UPRE High correlation between σ for both χ 2 (GSVD and LSQR) obtained. p values at or near 1. Algorithms converge with very few regularization calculations, average 7. The bidiagonalized system is on average much smaller than the system size. Errors distributions equivalent. Total cost of LSQR is the cost of bidiagonalization plus use of bidiagonalization to obtain a solution, eg column 2 plus column 4 solves. Errors of UPRE are similar to χ 2 approaches. UPRE fails more often ( more discarded solutions with high noise). MATHEMATICS AND STATISTICS 39 / 40

45 THANK YOU! MATHEMATICS AND STATISTICS 40 / 40