Towards Improved Sensitivity in Feature Extraction from Signals: one and two dimensional

Transcription

1 Towards Improved Sensitivity in Feature Extraction from Signals: one and two dimensional Rosemary Renaut, Hongbin Guo, Jodi Mead and Wolfgang Stefan Supported by Arizona Alzheimer s Research Center and NIH Arizona State University August 2007 Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 1 / 56

2 Outline 1 PET Images 2 Image Restoration Optimization for Regularized Least Squares Blind deconvolution Regularized TLS A Newton Algorithm Scaled Total Least Squares Numerical Results 3 Examples 4 Chi squared Method Background Algorithm Single Variable Newton Method Extend for General D: Generalized Tikhonov 5 Conclusions and Future Work Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 2 / 56

3 Outline 1 PET Images 2 Image Restoration Optimization for Regularized Least Squares Blind deconvolution Regularized TLS A Newton Algorithm Scaled Total Least Squares Numerical Results 3 Examples 4 Chi squared Method Background Algorithm Single Variable Newton Method Extend for General D: Generalized Tikhonov 5 Conclusions and Future Work Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 2 / 56

4 Outline 1 PET Images 2 Image Restoration Optimization for Regularized Least Squares Blind deconvolution Regularized TLS A Newton Algorithm Scaled Total Least Squares Numerical Results 3 Examples 4 Chi squared Method Background Algorithm Single Variable Newton Method Extend for General D: Generalized Tikhonov 5 Conclusions and Future Work Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 2 / 56

5 Outline 1 PET Images 2 Image Restoration Optimization for Regularized Least Squares Blind deconvolution Regularized TLS A Newton Algorithm Scaled Total Least Squares Numerical Results 3 Examples 4 Chi squared Method Background Algorithm Single Variable Newton Method Extend for General D: Generalized Tikhonov 5 Conclusions and Future Work Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 2 / 56

6 Outline 1 PET Images 2 Image Restoration Optimization for Regularized Least Squares Blind deconvolution Regularized TLS A Newton Algorithm Scaled Total Least Squares Numerical Results 3 Examples 4 Chi squared Method Background Algorithm Single Variable Newton Method Extend for General D: Generalized Tikhonov 5 Conclusions and Future Work Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 2 / 56

7 Example of typical PET scan Typical PET Images show High noise content (non Gaussian) High blurring Partial volume Effects Reconstruction artifacts Reconstruction using filtered backprojection Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 3 / 56

8 Long Term Goals/Methods of the Study Improved Sensitivity for identifying features in images: Signal restoration one dimensional Signal restoration two dimensional Deblurring Use information extracted from images for assisting with Alzheimer s Disease diagnosis. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 4 / 56

9 More About PET images Recall blurring of standard PET images: Blurring Model Spatially invariant blur: g = f h + n g - recorded image h - point spread function(psf) n - noise f - desired image Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 5 / 56

10 Inverse Problem Goal Given g find f Problem is ill-posed. We cannot invert to find f. Regularization is needed. ˆf = arg min { g f h λr(f )} f R(f ) is a regularization term which picks a best image according to desired properties. Difficulty Changing the contrast can be a difficulty for quantification from images. Benefit May show anatomical change without quantification. Important to identify edges. Difficulty What is λ? Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 6 / 56

11 Regularization Methods (short overview) Common methods are Tikhonov (TK). R(f ) = TK(f ) = f (x) 2 dx. Total Variation (TV) R(f ) = TV(f ) = Ω Ω f (x) dx. Sparse deconvolution (L 1 ) (not relevant for PET images) R(f ) = f 1 = f (x) dx. Ω Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 7 / 56

12 Optimization Objective function of the data fit term is convex TK is a linear least squares (LS) problem ˆf = arg min f { g Hf λ f 2 2 )} The TV objective function is non differentiable J(f ) = g Hf λ f 1 Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 8 / 56

13 Differentiability of TV - 1D (tensor product in 2D) R(f ) = i f i+1 f i For a small β define R β = i (f i+1 f i ) 2 + β choose β in 10 5 to 10 9 Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 9 / 56

14 Parameter Dependence: characteristics of solution ˆf = arg min { g f h λr(f )} f λ Governs the trade off between the fit to the data and the smoothness of the reconstruction and can be picked by the L-curve approach, (see Hansen, Inverse Problems) : PERHAPS? TK yields a smooth reconstruction. But only depends on λ. PRO : TV yields a piece wise constant reconstruction and preserves the edges of the image. CON But depends on λ and β L1 yields spike trains: also depends on λ and β. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 10 / 56

15 Comparing TV and TK Regularization Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 11 / 56

16 Numerical optimization scheme: to handle regularization To find the minimum we use a limited memory BFGS (see Vogel, Computational Methods for Inverse Problems and Nocedal) A quasi Newton Method where the estimated Hessian in each step is updated by a rank 2 update. Only a limited number of update vectors are kept, e.g. 10. Evaluation of the OF and its gradient is cheap (some FFTs and sparse matrix-vector multiplications) Problems are usually large and many iterations are needed. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 12 / 56

17 Point Spread Function The PSF is usually unknown or only estimated Estimates exist for PET scanners from phantom scans Actually PSF is spatially variant and also depends on the scanned object Hence even if provided PSF is always only an estimate For the PET scans presented here, a 6mm half width Gaussian was assumed. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 13 / 56

18 Simulated PET On the left simulated PET from blurred segmented MRI scan using Gaussian PSF and noise added. On the right deblurred PET with TV and known PSF. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 14 / 56

19 Recover real PET image Reconstruction done using Filtered Back Projection PSF estimated by a Gaussian TV regularization Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 15 / 56

20 Observations Image improvement is possible even with a rough estimation of the PSF (non-blind deconvolution) Total Variation regularization (piecewise constant solution) is appropriate: intensity levels depend on the tissue type. Improvement requires better approximation of the PSF Blind Deconvolution? Total Least Squares Increased Artifacts and noise. (More post processing can improve this) Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 16 / 56

21 Blind deconvolution h is usually unknown Blind deconvolution solves min f,h f h g λ 1 L 1 (f f 0 ) p1 + λ 2 L 2 (h h 0 ) p2 very limited uniqueness results for p 1 = p 2 = 2 and L 1 = L 2 = I e.g. by Scherzer and Justen For general L and p, no uniqueness, In practical applications p = 1 is often better, Stefan Practical applications - we should know something about h Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 17 / 56

22 Total least squares (TLS) - when we know something about the PSF Rewrite convolution as matrix vector product: g = Hf + n H is a Toeplitz matrix of Point Spread Function TLS assumes error in H and g i.e. g = (H + E)f + n Total least squares solution f TLS solves min E n 2 F subject to g = (H + E)f + n Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 18 / 56

23 History: Total Least Squares Statistics: Errors in variables regression(gleser, 1981) Measurement error models (Fuller, 1987) Orthogonal distance regression (Adcock, 1878) Generalized least squares (Madansky, 1959,... ) Numerical Analysis: Golub & van Loan, System Identification: Global linear least squares (Staar, 1981) eigenvector method (Levin, 1964) Koopmans-Levin method (Fernando & Nicholson, 1985) Compensated least squares (Stoica & Soderstrom, 1982) Signal Processing: Minimum norm method (Kumaresan &Tufts, 1983) Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 19 / 56

24 min (E, f ) F subject to (A + E)x = b + f Classical Algorithm Golub Reinsch Van-Loan [ ] xtls Solve [Â ˆb] = 0, Â = A + E, 1 ˆb = b + f. SVD Solution uses the SVD of augmented matrix [A b]. [ ] xtls = 1 v 1 n+1. v n+1,n+1 Direct Compute the SVD and solve. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 20 / 56

25 Rayleigh Quotient Formulation for TLS An Iterative Approach: x TLS = argmin x φ(x) = argmin x Ax b x 2, φ is the Rayleigh quotient of matrix [A, b]. TLS minimizes the sum of squared normalized residuals. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 21 / 56

26 Tikhonov TLS Regularization Stabilize TLS with realistic constraint on data Lx δ. δ is prescribed from known information on the solution, and L is typically a differential operator. Reformulation with Lagrange Multiplier If the constraint is active the solution x satisfies normal equations (A T A + λ I I + λ L L T L)x = A T b, (x ) T L T Lx δ 2 = 0, λ I = Ax b x 2, λ L = λ(1 + x 2 ), λ = 1 δ 2 ( b T (Ax b) 1 + x 2 + Ax b 2 (1 + x 2 ) 2 ), Golub, Hansen and O Leary, Solution technique is parameter dependent: λ L, λ I, δ, λ Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 22 / 56

27 Rayleigh Quotient Formulation for RTLS Recall x TLS = argmin x φ(x) = argmin x Ax b x 2, Formulate regularization for TLS in RQ form min x φ(x) subject to Lx δ. Augmented Lagrangian L(x, λ) = φ(x) + λ( Lx 2 δ 2 ). Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 23 / 56

28 Eigenvalue Problem RTLS: Renaut & Guo (SIAM 2004) ( B(x x ) 1 ) = λ I ( x 1 ), ( B(x A ) = T A + λ L (x )L T L, A T b b T A, λ L (x )γ + b T b ), λ L = λ(1 + x 2 ), Seek the minimal eigenpair for B. Note B is not constant. Specify constraint Lx 2 δ 2, use γ = δ 2 or Lx 2 This is a Parameter Dependent Eigenvalue Problem Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 24 / 56

29 Development At a solution constraint is active: measure normalized discrepancy Denote B(θ) = M + θn, N = g(x) = ( Lx 2 δ 2 )/(1 + x 2 ) ( L T L 0 0 δ 2 Denote eigenpair for smallest eigenvalue of B(θ) as ϱ n+1, (x T θ, 1)T Reformulation: ). For a constant δ, find a θ such that g(x θ ) = 0. A Newton iteration. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 25 / 56

30 The Update Equation for θ = λ L λ (k+1) L = λ (k) (k) λ(k) L L + ι δ 2 g(x λ (k) ), L 0 < ι (k) 1 such that g(x (k+1) λ )g(x (0) L λ ) > 0. L Theorem: Given λ (0) L > 0 iteration converges to unique solution, λ L. Observe: B(λ (k) L ) is always positive definite. For 0 < λ(0) L < θ c iteration is monotically increasing or decreasing dependent on λ (0) L >< λ L. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 26 / 56

31 EXACT RTLS: Alternating Iteration on λ L and x. Algorithm Given δ, λ (0) L > 0, calculate eigenpair (ϱ (0) n+1, x (0) ). Set k = 0. Update λ (k) L and x (k) until convergence. 1 While not converged Do ι (k) = 1 1 Inner Iteration: Until g(x (k+1) )g(x (0) ) > 0 1 Update λ (k+1) L., and corresponding eigenvector, 2 Calculate smallest eigenvalue, ϱ (k) n+1 [(x (k+1) ) T, 1] T, of matrix B(λ (k) L ). 3 If g(x (k+1) )g(x (0) ) < 0, set ι (k) = ι (k) /2 else Break. 2 Test for convergence. If converged Break else k = k End Do. x RTLS = x (k). Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 27 / 56

32 Computational Considerations: Generalized SVD Choice of γ: The theory is developed for γ = δ 2 but the algorithm can use γ = Lx k,j 2. If blow up does not occur, the algorithm converges more quickly. All approaches need to solve normal equations Jw = f, (A T A + λ L L T L)w = f GSVD For the augmented matrix [A, L], 2m 2 n + 15n 3 flops. ( ) Σ 0 A = U X 1, L = V ( M 0 ) X 1. 0 I n p U, V, m n and p p, resp., orthonormal. X, n n nonsingular.σ, M diag., p p, entries σ i λ i, resp. Then [( ) ( )] Σ 2 0 M λ 0 I L X 1 w = X T f. n p 0 0 Efficient Direct solution can be found efficiently 8n 2 flops. Solve triangular systems for each λ L. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 28 / 56

33 Generalization: Scaled TLS Theory (Paige and Strakos, Numerische Mathematik) min E n 2 F subject to g = (H + E)f + n γ Minimum is obtained as the minimum singular value of [H, γg] For flexibility use Rayleigh quotient formulation γ = 0 is the LS problem min f Hf g γ 2 f 2 2 γ = 1 is the standard TLS problem γ accounts for different noise levels in H and g. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 29 / 56

34 Scaled Total Least Squares with Regularization Regularize scaled RQ: Hf g 2 2 min f 1 + γ 2 f 2 + λr(f ) 2 Permits careful investigation of effect of noise levels in H and g. Which is greater, the error in the PSF or the error in the measured data? Can be solved using linear algebra methods, but only if TK regularization. Use BFGS methods Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 30 / 56

35 Test Problem Noisy Shepp Logan Phantom Shepp Logan Phantom Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 31 / 56

36 Test Problem Noisy Shepp Logan Phantom x 2 e 2σ 2 Blur with Gaussian h(x) = 1 σ 2π width) Take forward Radon transform with 45 angles Add Poisson Noise to sinogram Transform back, with filtered back projection with σ = 1.5 (6mm half Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 31 / 56

37 Deconvolving the Shepp-Logan Phantom Gauss PSF with σ = 2 and TV regularization γ 2 = 0 γ 2 = 7.3e 9 γ 2 = 3.7e 7 Scaling shows how to improve impact of badly chosen PSF. Scaling Hf = g = 1 (Notice for given image g ) For scaled problem γ 2 is 1, 50, resp. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 32 / 56

38 Real PET data Real PET data Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 33 / 56

39 Numerical Results Scaled RTVTLS Use a PSF with 6mm half width Gaussian and then restore γ = 0 Least Squares Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 34 / 56

40 Numerical Results Scaled RTVTLS Use a PSF with 6mm half width Gaussian and then restore γ = 3.7e 7 Total Least Squares Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 34 / 56

41 Numerical Results Scaled RTVTLS Use a PSF with 6mm half width Gaussian and then restore γ = 1.5e 5 Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 34 / 56

42 Numerical Results Scaled RTVTLS Use a PSF with 6mm half width Gaussian and then restore γ = 1e 4 Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 34 / 56

43 Observations RTLS with TV handles inexact PSFs better then simple RLS Scaled RTVTLS with parameter γ in Hf g 2 2 min f 1 + γ 2 f 2 + λr(f ) 2 allows further tuning in case of an unknown PSF Iterations are expensive. All methods are very parameter dependent. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 35 / 56

44 Example : Regularization parameter by L-curve Notice optimal λ by L-curve is similar for LS and Scaled TLS. But the corner of the L-curve is not well defined. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 36 / 56

45 The L-curve Calculating L-curve is expensive. Other methods for finding the regularization parameter; GCV, UPRE, A new method: Use Statistical Information Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 37 / 56

46 Regularized Least Squares: Using statistical information Rewrite regularized weighted least squares (Tikhonov) ˆx = argmin J(x) = argmin{ Ax b W 2 b + x x 0 W 2 x }. (1) Typically W b is inverse covariance matrix for data b. Generalized Tikhonov ˆx = argmin J(x) = argmin{ Ax b W 2 b + D(x x 0 ) 2 W x }. (2) Assume N (A) N (D) = Typically W x = λ 2 I, λ unknown penalty parameter Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 38 / 56

47 Standard Methods: Generalized Cross-Validation (GCV) 1 Estimates predictive risk. 2 No statistical information 3 Minimize GCV function b Ãx(λ) 2 2 [trace(i m A(W x ))] 2, W x = λ 2 I n 4 Expensive - need to find a range of λ values. 5 Uses GSVD to make calculations more efficient. 6 Minimum needed Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 39 / 56

48 Standard Methods: Generalized Cross-Validation (GCV) 1 Estimates predictive risk. 2 No statistical information 3 Minimize GCV function b Ãx(λ) 2 2 [trace(i m A(W x ))] 2, W x = λ 2 I n 4 Expensive - need to find a range of λ values. 5 Uses GSVD to make calculations more efficient. 6 Multiple minima Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 39 / 56

49 Standard Methods: Generalized Cross-Validation (GCV) 1 Estimates predictive risk. 2 No statistical information 3 Minimize GCV function b Ãx(λ) 2 2 [trace(i m A(W x ))] 2, W x = λ 2 I n 4 Expensive - need to find a range of λ values. 5 Uses GSVD to make calculations more efficient. 6 Sometimes flat Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 39 / 56

50 Standard Methods Unbiased Predictive Risk Estimation (UPRE) 1 Minimize expected value of predictive risk 2 Uses statistical information 3 Minimize UPRE function b Ax(λ) 2 W b +2 trace(a(w x )) m 4 Expensive - need to find a range of λ values. 5 Uses GSVD to make calculations more efficient. 6 Minimum needed Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 40 / 56

51 phillips Fredholm integral equation (Hansen) Example Error 10% 1 Add noise to b 2 Standard deviation σ bi =.01 b i +.1b max 3 Covariance matrix C b = σ 2 b I m 4 σ 2 b average of σ2 b i Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 41 / 56

52 phillips Fredholm integral equation (Hansen) L-curve Solution 1 Add noise to b 2 Standard deviation σ bi =.01 b i +.1b max 3 Covariance matrix C b = σ 2 b I m 4 σ 2 b average of σ2 b i 5 Uncertainty estimates shown Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 41 / 56

53 phillips Fredholm integral equation (Hansen) GCV Solution 1 Add noise to b 2 Standard deviation σ bi =.01 b i +.1b max 3 Covariance matrix C b = σ 2 b I m 4 σ 2 b average of σ2 b i 5 Uncertainty estimates shown Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 41 / 56

54 phillips Fredholm integral equation (Hansen) UPRE Solution 1 Add noise to b 2 Standard deviation σ bi =.01 b i +.1b max 3 Covariance matrix C b = σ 2 b I m 4 σ 2 b average of σ2 b i 5 Uncertainty estimates shown Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 41 / 56

55 phillips Fredholm integral equation (Hansen) χ 2 curve Solution 1 Add noise to b 2 Standard deviation σ bi =.01 b i +.1b max 3 Covariance matrix C b = σ 2 b I m 4 σ 2 b average of σ2 b i 5 Uncertainty estimates shown Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 41 / 56

56 phillips Fredholm integral equation (Hansen) Comparison Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 42 / 56

57 blur Atmospheric (Gaussian PSF) (Hansen): Again with noise Solution on Left and Degraded on the Right Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 43 / 56

58 blur Atmospheric (Gaussian PSF) (Hansen): Again with noise Solutions using x 0 = 0, Generalized Tikhonov Second Derivative 5% noise Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 43 / 56

59 blur Atmospheric (Gaussian PSF) (Hansen): Again with noise olutions using x 0 = 0, Generalized Tikhonov Second Derivative 10% noise Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 43 / 56

60 General Result: Tikhonov Cost functional is χ 2 r.v. Theorem (Rao:73, Tarantola, Mead (2007)) J(x) = (b Ax) T C b 1 (b Ax) + (x x 0 ) T C x 1 (x x 0 ), x and b are stochastic (need not be normal) r = b Ax 0 are iid. Matrices C b and C x are SPD - are considered as covariance matrices but need not be Then for large m, minimium value of J is a random variable it follows a χ 2 distribution with m degrees of freedom. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 44 / 56

61 Implication: Find W x such that J is χ 2 rv. Theorem implies m 2z α/2 < J(ˆx) < m + 2z α/2 for confidence interval (1 α). Equivalently m 2z α/2 < r T (AC x A T + C b ) 1 r < m + 2z α/2. Having found W x posterior inverse covariance matrix is W x = A T W b A + W x Note that W x is completely general Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 45 / 56

62 A New Algorithm for Estimating Model Covariance Algorithm (Mead 07) Given confidence interval parameter α, initial residual r = b Ax 0 and estimate of the data covariance C b, find L x which solves the nonlinear optimization. Minimize L x L T x 2 F Subject to m 2z α/2 < r T (AL x L T x A T + C b ) 1 r < m + 2z α/2 AL x L T x A T + C b well-conditioned. Use Matlab: expensive Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 46 / 56

63 Single Variable Approach 1 Seek to make an efficient and practical algorithm 2 First consider the single variable case W x = σx 2 I, where note regularization parameter λ = 1/σ x. 3 Use SVD to implement U b Σ b Vb T = W b 1/2 A, svs σ 1 σ 2... σ p and define s = U b W 1/2 b r: 4 Find σ x such that m 2z α/2 < s T 1 diag( σi 2 σx )s < m + 2z α/2. 5 Equivalently, find σ 2 x such that F (σ x ) = s T 1 diag( 1 + σxσ 2 i 2 )s m = 0. Scalar Root Finding: Newton s Method Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 47 / 56

64 First: Extension to Generalized Tikhonov Define ˆx GTik = argminj D (x) = argmin{ Ax b 2 W b + D(x x 0 ) 2 W x }, (3) Theorem For large m, the minimium value of J D is a random variable which follows a χ 2 distribution with m n + p degrees of freedom. Proof. Use the Generalized Singular Value Decomposition for [W b 1/2 A, W x 1/2 D] Find W x to satisfy J D is χ 2 with m n + p degrees of freedom. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 48 / 56

65 Newton Root Finding W x = σx 2 I p GSVD of [W 1/2 b A, D] [ Υ A = U 0 m n n ] X T D = V [M, 0 p n p ]X T, γ i are the generalized singular values m = m n + p p i=1 s2 i δ γ i 0 m i=n+1 s2 i, s i = s i /(γi 2 σx 2 + 1), i = 1,..., p t i = s i γ i. Solve F = 0, where F (σ x ) = s T s m and F (σ x ) = 2σ x t 2 2. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 49 / 56

66 Observations Example F Initialization cost GCV, UPRE, L-curve, χ 2 all use GSVD (or SVD). Algorithm is cheap as compared to GCV, UPRE, L-curve. F is monotonic decreasing, even Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 50 / 56

67 Observations Example F Initialization cost GCV, UPRE, L-curve, χ 2 all use GSVD (or SVD). Algorithm is cheap as compared to GCV, UPRE, L-curve. Solution either exists and is unique for positive σ Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 50 / 56

68 Observations Insufficient degrees of freedom Initialization cost GCV, UPRE, L-curve, χ 2 all use GSVD (or SVD). Algorithm is cheap as compared to GCV, UPRE, L-curve. Or no solution exists F (0) < 0. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 50 / 56

69 Some Generalized Tikhonov Solutions: First Order Derivative No Statistical Information C b is constant multiple of I C b = diag(σ 2 b i ) Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 51 / 56

70 Some Generalized Tikhonov Solutions: x 0 = 0: Exponential noise No Statistical Information C b is constant multiple of I C b = diag(σ 2 b i ) Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 52 / 56

71 Conclusions χ 2 Newton algorithm is cost effective- converges in 5 10 iterations. It performs as well ( or better) than GCV and UPRE when statistical information is available. Should be method of choice when statistical information is provided Method can be adapted to find W b if W x is provided. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 53 / 56

72 Conclusions χ 2 Newton algorithm is cost effective- converges in 5 10 iterations. It performs as well ( or better) than GCV and UPRE when statistical information is available. Should be method of choice when statistical information is provided Method can be adapted to find W b if W x is provided. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 53 / 56

73 Conclusions χ 2 Newton algorithm is cost effective- converges in 5 10 iterations. It performs as well ( or better) than GCV and UPRE when statistical information is available. Should be method of choice when statistical information is provided Method can be adapted to find W b if W x is provided. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 53 / 56

74 Conclusions χ 2 Newton algorithm is cost effective- converges in 5 10 iterations. It performs as well ( or better) than GCV and UPRE when statistical information is available. Should be method of choice when statistical information is provided Method can be adapted to find W b if W x is provided. Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 53 / 56

75 Future Work Further investigation of RTVTLS and relation to RTVLS (also with scaling) Improve efficiency of algorithms (methods of Guo and Renaut) Extend the χ 2 method for RTLS schemes, to make methods more efficient. Extend to use with TV regularization? But the theory does not apply? What can be done? Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 54 / 56

76 collaborators Hongbin Guo - Total Least Squares Kewei Chen for the data and discussions on PET imaging Wolfgang Stefan - Total Variation Minimization Jodi Mead - parameter estimation. Supported by: Arizona Alzheimer s Research Center, NIH NIBIB and NSF Renaut, Guo, Mead and Stefan (ASU) Signal Restoration Bulgaria 55 / 56

77 THANK YOU!