Newton s Method for Estimating the Regularization Parameter for Least Squares: Using the Chi-curve
|
|
- Chad Hill
- 5 years ago
- Views:
Transcription
1 Newton s Method for Estimating the Regularization Parameter for Least Squares: Using the Chi-curve Rosemary Renaut, Jodi Mead Arizona State and Boise State Copper Mountain Conference on Iterative Methods April 2008
2 Outline 1 Introduction Finding the Regularization Parameter 2 A Statistically based method: Chi squared Method Background Algorithm Single Variable Newton Method Extend for General D: Generalized Tikhonov Observations 3 Avoiding the GSVD - Solution for Large Scale Problems 4 Results 5 Future Work 6 Other Results
3 Regularized Least Squares for Ax = b Consider discrete ill-posed systems: A R m n, b R m, x R n Ax = b + e, e is the m vector of random measurement errors with mean 0 and positive definite covariance matrix C b = E(ee T ). Assume that C b is known. For uncorrelated measurements C b is diagonal matrix of standard deviations of the errors. For correlated measurements, let W b = C b 1 and L b L b T = W b be the Choleski factorization of W b and weight the equation: L b Ax = L b b + ẽ, ẽ are uncorrelated.
4 The General Problem: Weighted Regularized Least Squares Formulation ˆx = argmin J(x) = argmin{ Ax b 2 W b + x x 0 2 W x }. (1) x 0 is a reference solution, often x 0 = 0. Standard: W x = λi, λ unknown penalty parameter. Statistically, W x is inverse covariance matrix for the model x i.e. λ = 1/σ 2 x, σ 2 x the common variance in x. Assumes the resulting estimtates are uncorrelated. Question What is the correct λ? More generally, what is the correct W x?
5 The General Case : Tikhonov Regularization with Solution Mapping Formulation Generalized Tikhonov regularization, operator D acts on x. ˆx = argmin J(x) = argmin{ Ax b 2 W b + D(x x 0 ) 2 W x }. (2) Assume N (A) N (D) = Then solutions depend on λ : ˆx(λ) = argmin J(x) = argmin{ Ax b 2 W b + λ D(x x 0 ) 2 }. (3) Goal Can we estimate λ efficiently when W b is known?
6 Standard Methods 1 L-curve - trades off contributions of the residual and the regularization. 2 Generalized Cross-Validation (GCV)- (Statistical) Minimizes which estimates predictive risk. b Ax(λ) 2 W b [trace(i m A(λ))] 2, 3 Unbiased Predictive Risk Estimator (UPRE) - Statistical - Minimize expected value of predictive risk: b Ax(λ) 2 W b + 2 trace(a(λ)) m Problem Require generally multiple solutions Expensive to calculate Not always sufficient - might not converge to minimum, multiple minima
7 Some New Developments 1 Residual Periodogram ( O Leary and Rust) 2 The core problem reduction (Page, Strakos, Hnetynkova) 3 Projection Methods (Hansen, Kilmer, Espanol) 4 HyBR Nagy - Copper Mountain A χ 2 Method (Mead) - designed to find W x not just λ We focus on the method of Mead for its simplicity for one parameter and its potential to apply for general covariance matrix W x.
8 General Result: Tikhonov (D = I) Cost functional at minimum is χ 2 r.v. Theorem (Rao:73, Tarantola, Mead (2007)) J(x) = (b Ax) T C b 1 (b Ax) + (x x 0 ) T C x 1 (x x 0 ), x and b are stochastic (need not be normal) r = b Ax 0 are iid. (Assume no components are zero) Matrices C b = W b 1 and C x = W x 1 are SPD - Then for large m, minimium value of J is a random variable it follows a χ 2 distribution with m degrees of freedom.
9 Implication: Find W x such that J is χ 2 r.v. Theorem implies m 2z α/2 < J(ˆx) < m + 2z α/2 for confidence interval (1 α), ˆx the solution. Equivalently, when D = I, m 2z α/2 < r T (AC x A T + C b ) 1 r < m + 2z α/2. Having found W x posterior inverse covariance matrix is W x = A T W b A + W x Note no assumptions on W x : it is completely general
10 An Illustrative Example: phillips Fredholm integral equation (Hansen) Example Error 10% 1 Add noise to b 2 Covariance matrix C b = σ 2 b I m = W b 1 3 is the original b and noisy data.
11 An Illustrative Example: phillips Fredholm integral equation (Hansen) 1 Compare Solutions: 2 + is reference x 0. is exact. 3 L-Curve o 4 Three other solutions: UPRE, GCV and χ 2 method (blue, magenta, black) 5 Each method gives different solution - but UPRE, GCV and χ 2 are comparable. Comparison with new method
12 Newton s method to find the regularization parameter 1 Let W x = σ 2 x I, where regularization parameter λ = 1/σ 2 x. 2 Use SVD to implement U b Σ b Vb T = W b 1/2 A, svs σ 1 σ 2... σ p and define s = U b W 1/2 b r: 3 Find σ x such that m 2z α/2 < s T 1 diag( σi 2 σx )s < m + 2z α/2. 4 Or find σ 2 x such that F(σ x ) = s T 1 diag( 1 + σxσ 2 i 2 )s m = 0. Scalar Root Finding: Newton s Method
13 Extension to Tikhonov with Mapping D ˆx GTik = argminj D (x) = argmin{ Ax b 2 W b + D(x x 0 ) 2 W x }, (4) Theorem For large m, the minimium value of J D is a random variable which follows a χ 2 distribution with m n + p degrees of freedom. (Assuming that no components of r are zero) Proof. Generalized Singular Value Decomposition for [W b 1/2 A, W x 1/2 D] Goal Find W x such that J D is χ 2 with m n + p degrees of freedom
14 Newton Root Finding W x = σx 2 I p GSVD of [W 1/2 b A, D] [ Υ A = U 0 m n n ] X T D = V [M, 0 p n p ]X T, γ i are the generalized singular values m = m n + p p i=1 s2 i δ γ i 0 m i=n+1 s2 i, s i = s i /(γi 2 σx 2 + 1), i = 1,..., p t i = s i γ i. Formulation Find root of p i=1 ( 1 γi 2 σx+1 2 )s2 i + m i=n+1 s2 i Solve F = 0, where = m F(σ x ) = s T s m and F (σ x ) = 2σ x t 2 2.
15 Relationship to Discrepancy Principle The discrepancy principle can be implemented by a Newton method. Finds σ x such that the regularized residual satisfies σ 2 b = 1 m b Ax(σ) 2 2. (5) Consistent with our notation p 1 ( γi 2 σ )2 si 2 + i=1 m i=n+1 s 2 i = m, (6) Weight in the first sum is squared here, otherwise functional is the same. But discrepancy principle often oversmooths. What happens here?
16 Some Generalized Tikhonov Solutions: First Order Derivative No Statistical Information Solution is Smoothed C b = diag(σ 2 b i ) Solution is not smoothed
17 Discussion F is monotonic decreasing Solution either exists and is unique for positive σ Or no solution exists F(0) < 0. implies incorrect scaling of the model Theoretically, lim σ F > 0 possible. Equivalent to λ = 0. No regularization needed.
18 A Newton Iteration without the GSVD: Large Scale Problems Evaluate J D as a function of λ. Let r = W b 1 2 r J D (λ) = r 2 r T A(λ) r, where A(λ) = A(A T W b A + λd T D) 1 A T is the influence matrix. Let y(λ) be the solution for x 0 = 0 then J D (λ) = Dy(λ) 2 0, unless y N (D) To find σ = 1/ λ we use the Newton Iteration σ k+1 = σ k ( ( σk Dy )2 (J D (σ k ) m)).
19 Experiments Take example from Hansen s toolbox, eg shaw, phillips. Generate 100 copies for each noise level.001,.005,.01,.05,.1. Solve for 100 cases using GSVD and inexact Newton with increasing number of inner iterations. Do pairwise t test on the obtained σ to determine dependence on inexact solves Repeat for seismic data sets, 48 signals. Check p values.
20 Summary of Results 1 High correlation between σ for both Newton Methods is obtained. p values at or near 1. 2 Inexact solves can be used. eg Inner Noise Iterations Table: p values for the uncorrelated case solving Shaw, for size 64.
21 An Example : Comparing Newton Solutions for Real Data
22 An Example of Comparison with L-curve and UPRE Solutions Observation UPRE and L-curve underestimate λ.
23 Future Plans 1 Make Newton more robust/efficient - note the difficulty with the asymptote in F. Use inverse interpolation? Recycle subspace information in calculation of iterative solutions. Extend for image deblurring. 2 Edge preserving regularization. 3 Diagonal Weighting schemes
24 Example: Seismic Signal Restoration Real data set of 48 signals of length The point spread function is derived from the signals Solve Pf = g, where P is psf matrix, g is signal and restore f. Calculate the signal variance pointwise over all 48 signals.
25 Tikhonov Regularization
26 Comparing Newton Solutions
27 Comparing Newton Solutions
Regularization Parameter Estimation for Least Squares: A Newton method using the χ 2 -distribution
Regularization Parameter Estimation for Least Squares: A Newton method using the χ 2 -distribution Rosemary Renaut, Jodi Mead Arizona State and Boise State September 2007 Renaut and Mead (ASU/Boise) Scalar
More informationNewton s method for obtaining the Regularization Parameter for Regularized Least Squares
Newton s method for obtaining the Regularization Parameter for Regularized Least Squares Rosemary Renaut, Jodi Mead Arizona State and Boise State International Linear Algebra Society, Cancun Renaut and
More informationThe Chi-squared Distribution of the Regularized Least Squares Functional for Regularization Parameter Estimation
The Chi-squared Distribution of the Regularized Least Squares Functional for Regularization Parameter Estimation Rosemary Renaut DEPARTMENT OF MATHEMATICS AND STATISTICS Prague 2008 MATHEMATICS AND STATISTICS
More informationThe Chi-squared Distribution of the Regularized Least Squares Functional for Regularization Parameter Estimation
The Chi-squared Distribution of the Regularized Least Squares Functional for Regularization Parameter Estimation Rosemary Renaut Collaborators: Jodi Mead and Iveta Hnetynkova DEPARTMENT OF MATHEMATICS
More informationStatistically-Based Regularization Parameter Estimation for Large Scale Problems
Statistically-Based Regularization Parameter Estimation for Large Scale Problems Rosemary Renaut Joint work with Jodi Mead and Iveta Hnetynkova March 1, 2010 National Science Foundation: Division of Computational
More informationA Hybrid LSQR Regularization Parameter Estimation Algorithm for Large Scale Problems
A Hybrid LSQR Regularization Parameter Estimation Algorithm for Large Scale Problems Rosemary Renaut Joint work with Jodi Mead and Iveta Hnetynkova SIAM Annual Meeting July 10, 2009 National Science Foundation:
More informationTowards Improved Sensitivity in Feature Extraction from Signals: one and two dimensional
Towards Improved Sensitivity in Feature Extraction from Signals: one and two dimensional Rosemary Renaut, Hongbin Guo, Jodi Mead and Wolfgang Stefan Supported by Arizona Alzheimer s Research Center and
More informationTHE χ 2 -CURVE METHOD OF PARAMETER ESTIMATION FOR GENERALIZED TIKHONOV REGULARIZATION
THE χ -CURVE METHOD OF PARAMETER ESTIMATION FOR GENERALIZED TIKHONOV REGULARIZATION JODI MEAD AND ROSEMARY A RENAUT Abstract. We discuss algorithms for estimating least squares regularization parameters
More informationA Newton Root-Finding Algorithm For Estimating the Regularization Parameter For Solving Ill- Conditioned Least Squares Problems
Boise State University ScholarWorks Mathematics Faculty Publications and Presentations Department of Mathematics 1-1-2009 A Newton Root-Finding Algorithm For Estimating the Regularization Parameter For
More informationSIGNAL AND IMAGE RESTORATION: SOLVING
1 / 55 SIGNAL AND IMAGE RESTORATION: SOLVING ILL-POSED INVERSE PROBLEMS - ESTIMATING PARAMETERS Rosemary Renaut http://math.asu.edu/ rosie CORNELL MAY 10, 2013 2 / 55 Outline Background Parameter Estimation
More informationPreconditioning. Noisy, Ill-Conditioned Linear Systems
Preconditioning Noisy, Ill-Conditioned Linear Systems James G. Nagy Emory University Atlanta, GA Outline 1. The Basic Problem 2. Regularization / Iterative Methods 3. Preconditioning 4. Example: Image
More informationPreconditioning. Noisy, Ill-Conditioned Linear Systems
Preconditioning Noisy, Ill-Conditioned Linear Systems James G. Nagy Emory University Atlanta, GA Outline 1. The Basic Problem 2. Regularization / Iterative Methods 3. Preconditioning 4. Example: Image
More informationRegularization parameter estimation for underdetermined problems by the χ 2 principle with application to 2D focusing gravity inversion
Regularization parameter estimation for underdetermined problems by the χ principle with application to D focusing gravity inversion Saeed Vatankhah, Rosemary A Renaut and Vahid E Ardestani, Institute
More informationInverse Ill Posed Problems in Image Processing
Inverse Ill Posed Problems in Image Processing Image Deblurring I. Hnětynková 1,M.Plešinger 2,Z.Strakoš 3 hnetynko@karlin.mff.cuni.cz, martin.plesinger@tul.cz, strakos@cs.cas.cz 1,3 Faculty of Mathematics
More informationGolub-Kahan iterative bidiagonalization and determining the noise level in the data
Golub-Kahan iterative bidiagonalization and determining the noise level in the data Iveta Hnětynková,, Martin Plešinger,, Zdeněk Strakoš, * Charles University, Prague ** Academy of Sciences of the Czech
More informationRegularization Parameter Estimation for Underdetermined problems by the χ 2 principle with application to 2D focusing gravity inversion
Regularization Parameter Estimation for Underdetermined problems by the χ principle with application to D focusing gravity inversion Saeed Vatankhah, Rosemary A Renaut and Vahid E Ardestani, Institute
More informationRegularization methods for large-scale, ill-posed, linear, discrete, inverse problems
Regularization methods for large-scale, ill-posed, linear, discrete, inverse problems Silvia Gazzola Dipartimento di Matematica - Università di Padova January 10, 2012 Seminario ex-studenti 2 Silvia Gazzola
More informationParameter estimation: A new approach to weighting a priori information
c de Gruyter 27 J. Inv. Ill-Posed Problems 5 (27), 2 DOI.55 / JIP.27.xxx Parameter estimation: A new approach to weighting a priori information J.L. Mead Communicated by Abstract. We propose a new approach
More informationETNA Kent State University
Electronic Transactions on Numerical Analysis. Volume 28, pp. 49-67, 28. Copyright 28,. ISSN 68-963. A WEIGHTED-GCV METHOD FOR LANCZOS-HYBRID REGULARIZATION JULIANNE CHUNG, JAMES G. NAGY, AND DIANNE P.
More informationLEAST SQUARES PROBLEMS WITH INEQUALITY CONSTRAINTS AS QUADRATIC CONSTRAINTS
LEAST SQUARES PROBLEMS WITH INEQUALITY CONSTRAINTS AS QUADRATIC CONSTRAINTS JODI MEAD AND ROSEMARY A RENAUT. Introduction. The linear least squares problems discussed here are often used to incorporate
More informationInverse Theory. COST WaVaCS Winterschool Venice, February Stefan Buehler Luleå University of Technology Kiruna
Inverse Theory COST WaVaCS Winterschool Venice, February 2011 Stefan Buehler Luleå University of Technology Kiruna Overview Inversion 1 The Inverse Problem 2 Simple Minded Approach (Matrix Inversion) 3
More informationCOMPUTATIONAL ISSUES RELATING TO INVERSION OF PRACTICAL DATA: WHERE IS THE UNCERTAINTY? CAN WE SOLVE Ax = b?
COMPUTATIONAL ISSUES RELATING TO INVERSION OF PRACTICAL DATA: WHERE IS THE UNCERTAINTY? CAN WE SOLVE Ax = b? Rosemary Renaut http://math.asu.edu/ rosie BRIDGING THE GAP? OCT 2, 2012 Discussion Yuen: Solve
More informationNumerical Methods for Separable Nonlinear Inverse Problems with Constraint and Low Rank
Numerical Methods for Separable Nonlinear Inverse Problems with Constraint and Low Rank Taewon Cho Thesis submitted to the Faculty of the Virginia Polytechnic Institute and State University in partial
More informationLEAST SQUARES PROBLEMS WITH INEQUALITY CONSTRAINTS AS QUADRATIC CONSTRAINTS
LEAST SQUARES PROBLEMS WITH INEQUALITY CONSTRAINTS AS QUADRATIC CONSTRAINTS JODI MEAD AND ROSEMARY A RENAUT Abstract. Linear least squares problems with box constraints are commonly solved to find model
More informationestimator for determining the regularization parameter in
submitted to Geophys. J. Int. Application of the χ 2 principle and unbiased predictive risk estimator for determining the regularization parameter in 3D focusing gravity inversion Saeed Vatankhah, Vahid
More informationApplication of the χ 2 principle and unbiased predictive risk estimator for determining the regularization parameter in 3D focusing gravity inversion
Application of the χ principle and unbiased predictive risk estimator for determining the regularization parameter in 3D focusing gravity inversion arxiv:48.7v [math.na] 4 Aug 4 Saeed Vatankhah, Vahid
More informationUPRE Method for Total Variation Parameter Selection
UPRE Method for Total Variation Parameter Selection Youzuo Lin School of Mathematical and Statistical Sciences, Arizona State University, Tempe, AZ 85287 USA. Brendt Wohlberg 1, T-5, Los Alamos National
More informationDiscontinuous parameter estimates with least squares estimators
Discontinuous parameter estimates with least squares estimators J.L. Mead Boise State University, Department of Mathematicss, Boise, ID 8375-1555. Tel: 846-43, Fax: 8-46-1354 Email jmead@boisestate.edu.
More informationLanczos tridigonalization and Golub - Kahan bidiagonalization: Ideas, connections and impact
Lanczos tridigonalization and Golub - Kahan bidiagonalization: Ideas, connections and impact Zdeněk Strakoš Academy of Sciences and Charles University, Prague http://www.cs.cas.cz/ strakos Hong Kong, February
More informationREGULARIZATION PARAMETER SELECTION IN DISCRETE ILL POSED PROBLEMS THE USE OF THE U CURVE
Int. J. Appl. Math. Comput. Sci., 007, Vol. 17, No., 157 164 DOI: 10.478/v10006-007-0014-3 REGULARIZATION PARAMETER SELECTION IN DISCRETE ILL POSED PROBLEMS THE USE OF THE U CURVE DOROTA KRAWCZYK-STAŃDO,
More informationLaboratorio di Problemi Inversi Esercitazione 2: filtraggio spettrale
Laboratorio di Problemi Inversi Esercitazione 2: filtraggio spettrale Luca Calatroni Dipartimento di Matematica, Universitá degli studi di Genova Aprile 2016. Luca Calatroni (DIMA, Unige) Esercitazione
More informationSingular value analysis of Joint Inversion
Singular value analysis of Joint Inversion Jodi Mead Department of Mathematics Diego Domenzain Department of Geosciences James Ford Clearwater Analytics John Bradford Department of Geophysics Colorado
More informationAugmented Tikhonov Regularization
Augmented Tikhonov Regularization Bangti JIN Universität Bremen, Zentrum für Technomathematik Seminar, November 14, 2008 Outline 1 Background 2 Bayesian inference 3 Augmented Tikhonov regularization 4
More informationNumerical Linear Algebra and. Image Restoration
Numerical Linear Algebra and Image Restoration Maui High Performance Computing Center Wednesday, October 8, 2003 James G. Nagy Emory University Atlanta, GA Thanks to: AFOSR, Dave Tyler, Stuart Jefferies,
More informationComputational Methods. Eigenvalues and Singular Values
Computational Methods Eigenvalues and Singular Values Manfred Huber 2010 1 Eigenvalues and Singular Values Eigenvalues and singular values describe important aspects of transformations and of data relations
More informationLinear Inverse Problems
Linear Inverse Problems Ajinkya Kadu Utrecht University, The Netherlands February 26, 2018 Outline Introduction Least-squares Reconstruction Methods Examples Summary Introduction 2 What are inverse problems?
More informationECE295, Data Assimila0on and Inverse Problems, Spring 2015
ECE295, Data Assimila0on and Inverse Problems, Spring 2015 1 April, Intro; Linear discrete Inverse problems (Aster Ch 1 and 2) Slides 8 April, SVD (Aster ch 2 and 3) Slides 15 April, RegularizaFon (ch
More informationTikhonov Regularization of Large Symmetric Problems
NUMERICAL LINEAR ALGEBRA WITH APPLICATIONS Numer. Linear Algebra Appl. 2000; 00:1 11 [Version: 2000/03/22 v1.0] Tihonov Regularization of Large Symmetric Problems D. Calvetti 1, L. Reichel 2 and A. Shuibi
More informationMATH 3795 Lecture 10. Regularized Linear Least Squares.
MATH 3795 Lecture 10. Regularized Linear Least Squares. Dmitriy Leykekhman Fall 2008 Goals Understanding the regularization. D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least
More informationIll Posed Inverse Problems in Image Processing
Ill Posed Inverse Problems in Image Processing Introduction, Structured matrices, Spectral filtering, Regularization, Noise revealing I. Hnětynková 1,M.Plešinger 2,Z.Strakoš 3 hnetynko@karlin.mff.cuni.cz,
More informationMARN 5898 Regularized Linear Least Squares.
MARN 5898 Regularized Linear Least Squares. Dmitriy Leykekhman Spring 2010 Goals Understanding the regularization. D. Leykekhman - MARN 5898 Parameter estimation in marine sciences Linear Least Squares
More informationNON-NEGATIVELY CONSTRAINED LEAST
NON-NEGATVELY CONSTRANED LEAST SQUARES AND PARAMETER CHOCE BY THE RESDUAL PERODOGRAM FOR THE NVERSON OF ELECTROCHEMCAL MPEDANCE SPECTROSCOPY DATA Rosemary Renaut, Mathematics, ASU Jakob Hansen, Jarom Hogue
More informationLecture 1: Numerical Issues from Inverse Problems (Parameter Estimation, Regularization Theory, and Parallel Algorithms)
Lecture 1: Numerical Issues from Inverse Problems (Parameter Estimation, Regularization Theory, and Parallel Algorithms) Youzuo Lin 1 Joint work with: Rosemary A. Renaut 2 Brendt Wohlberg 1 Hongbin Guo
More informationThe Global Krylov subspace methods and Tikhonov regularization for image restoration
The Global Krylov subspace methods and Tikhonov regularization for image restoration Abderrahman BOUHAMIDI (joint work with Khalide Jbilou) Université du Littoral Côte d Opale LMPA, CALAIS-FRANCE bouhamidi@lmpa.univ-littoral.fr
More informationDiscrete ill posed problems
Discrete ill posed problems Gérard MEURANT October, 2008 1 Introduction to ill posed problems 2 Tikhonov regularization 3 The L curve criterion 4 Generalized cross validation 5 Comparisons of methods Introduction
More informationAdvanced Numerical Linear Algebra: Inverse Problems
Advanced Numerical Linear Algebra: Inverse Problems Rosemary Renaut Spring 23 Some Background on Inverse Problems Constructing PSF Matrices The DFT Rosemary Renaut February 4, 23 References Deblurring
More informationA MODIFIED TSVD METHOD FOR DISCRETE ILL-POSED PROBLEMS
A MODIFIED TSVD METHOD FOR DISCRETE ILL-POSED PROBLEMS SILVIA NOSCHESE AND LOTHAR REICHEL Abstract. Truncated singular value decomposition (TSVD) is a popular method for solving linear discrete ill-posed
More information10. Multi-objective least squares
L Vandenberghe ECE133A (Winter 2018) 10 Multi-objective least squares multi-objective least squares regularized data fitting control estimation and inversion 10-1 Multi-objective least squares we have
More informationNear-Optimal Spectral Filtering and Error Estimation for Solving Ill-Posed Problems
Near-Optimal Spectral Filtering and Error Estimation for Solving Ill-Posed Problems Viktoria Taroudaki Dianne P. O Leary May 1, 2015 Abstract We consider regularization methods for numerical solution of
More informationDedicated to Adhemar Bultheel on the occasion of his 60th birthday.
SUBSPACE-RESTRICTED SINGULAR VALUE DECOMPOSITIONS FOR LINEAR DISCRETE ILL-POSED PROBLEMS MICHIEL E. HOCHSTENBACH AND LOTHAR REICHEL Dedicated to Adhemar Bultheel on the occasion of his 60th birthday. Abstract.
More informationKrylov subspace iterative methods for nonsymmetric discrete ill-posed problems in image restoration
Krylov subspace iterative methods for nonsymmetric discrete ill-posed problems in image restoration D. Calvetti a, B. Lewis b and L. Reichel c a Department of Mathematics, Case Western Reserve University,
More informationunbiased predictive risk estimator for regularization
submitted to Geophys. J. Int. 3-D Projected L inversion of gravity data using truncated unbiased predictive risk estimator for regularization parameter estimation Saeed Vatankhah, Rosemary A. Renaut 2
More informationRegularizing inverse problems. Damping and smoothing and choosing...
Regularizing inverse problems Damping and smoothing and choosing... 141 Regularization The idea behind SVD is to limit the degree of freedom in the model and fit the data to an acceptable level. Retain
More informationExtension of GKB-FP algorithm to large-scale general-form Tikhonov regularization
Extension of GKB-FP algorithm to large-scale general-form Tikhonov regularization Fermín S. Viloche Bazán, Maria C. C. Cunha and Leonardo S. Borges Department of Mathematics, Federal University of Santa
More informationETNA Kent State University
Electronic Transactions on Numerical Analysis. Volume 31, pp. 204-220, 2008. Copyright 2008,. ISSN 1068-9613. ETNA NOISE PROPAGATION IN REGULARIZING ITERATIONS FOR IMAGE DEBLURRING PER CHRISTIAN HANSEN
More informationPROJECTED TIKHONOV REGULARIZATION OF LARGE-SCALE DISCRETE ILL-POSED PROBLEMS
PROJECED IKHONOV REGULARIZAION OF LARGE-SCALE DISCREE ILL-POSED PROBLEMS DAVID R. MARIN AND LOHAR REICHEL Abstract. he solution of linear discrete ill-posed problems is very sensitive to perturbations
More informationA Parameter-Choice Method That Exploits Residual Information
A Parameter-Choice Method That Exploits Residual Information Per Christian Hansen Section for Scientific Computing DTU Informatics Joint work with Misha E. Kilmer Tufts University Inverse Problems: Image
More informationSolving discrete ill posed problems with Tikhonov regularization and generalized cross validation
Solving discrete ill posed problems with Tikhonov regularization and generalized cross validation Gérard MEURANT November 2010 1 Introduction to ill posed problems 2 Examples of ill-posed problems 3 Tikhonov
More informationA Probability Review
A Probability Review Outline: A probability review Shorthand notation: RV stands for random variable EE 527, Detection and Estimation Theory, # 0b 1 A Probability Review Reading: Go over handouts 2 5 in
More informationThe Inversion Problem: solving parameters inversion and assimilation problems
The Inversion Problem: solving parameters inversion and assimilation problems UE Numerical Methods Workshop Romain Brossier romain.brossier@univ-grenoble-alpes.fr ISTerre, Univ. Grenoble Alpes Master 08/09/2016
More informationRegularization of the Inverse Laplace Transform with Applications in Nuclear Magnetic Resonance Relaxometry Candidacy Exam
Regularization of the Inverse Laplace Transform with Applications in Nuclear Magnetic Resonance Relaxometry Candidacy Exam Applied Mathematics, Applied Statistics, & Scientific Computation University of
More informationRegularization with Randomized SVD for Large-Scale Discrete Inverse Problems
Regularization with Randomized SVD for Large-Scale Discrete Inverse Problems Hua Xiang Jun Zou July 20, 2013 Abstract In this paper we propose an algorithm for solving the large-scale discrete ill-conditioned
More informationIterative Methods for Ill-Posed Problems
Iterative Methods for Ill-Posed Problems Based on joint work with: Serena Morigi Fiorella Sgallari Andriy Shyshkov Salt Lake City, May, 2007 Outline: Inverse and ill-posed problems Tikhonov regularization
More informationITERATIVE REGULARIZATION WITH MINIMUM-RESIDUAL METHODS
BIT Numerical Mathematics 6-3835/3/431-1 $16. 23, Vol. 43, No. 1, pp. 1 18 c Kluwer Academic Publishers ITERATIVE REGULARIZATION WITH MINIMUM-RESIDUAL METHODS T. K. JENSEN and P. C. HANSEN Informatics
More informationMulti-Linear Mappings, SVD, HOSVD, and the Numerical Solution of Ill-Conditioned Tensor Least Squares Problems
Multi-Linear Mappings, SVD, HOSVD, and the Numerical Solution of Ill-Conditioned Tensor Least Squares Problems Lars Eldén Department of Mathematics, Linköping University 1 April 2005 ERCIM April 2005 Multi-Linear
More informationNon-negatively constrained least squares and parameter choice by the residual periodogram for the inversion of electrochemical impedance spectroscopy
Non-negatively constrained least squares and parameter choice by the residual periodogram for the inversion of electrochemical impedance spectroscopy Jakob Hansen a, Jarom Hogue a, Grant Sander a, Rosemary
More informationDiscrete Ill Posed and Rank Deficient Problems. Alistair Boyle, Feb 2009, SYS5906: Directed Studies Inverse Problems 1
Discrete Ill Posed and Rank Deficient Problems Alistair Boyle, Feb 2009, SYS5906: Directed Studies Inverse Problems 1 Definitions Overview Inversion, SVD, Picard Condition, Rank Deficient, Ill-Posed Classical
More informationRegularization Parameter Selection Methods for Ill-Posed Poisson Maximum Likelihood Estimation
Regularization Parameter Selection Methods for Ill-Posed Poisson Maximum Likelihood Estimation Johnathan M. Bardsley and John Goldes Department of Mathematical Sciences University of Montana Missoula,
More informationOld and new parameter choice rules for discrete ill-posed problems
Old and new parameter choice rules for discrete ill-posed problems Lothar Reichel Giuseppe Rodriguez Dedicated to Claude Brezinski and Sebastiano Seatzu on the Occasion of Their 70th Birthdays. Abstract
More informationComparison of A-Posteriori Parameter Choice Rules for Linear Discrete Ill-Posed Problems
Comparison of A-Posteriori Parameter Choice Rules for Linear Discrete Ill-Posed Problems Alessandro Buccini a, Yonggi Park a, Lothar Reichel a a Department of Mathematical Sciences, Kent State University,
More informationMathematical Beer Goggles or The Mathematics of Image Processing
How Mathematical Beer Goggles or The Mathematics of Image Processing Department of Mathematical Sciences University of Bath Postgraduate Seminar Series University of Bath 12th February 2008 1 How 2 How
More information1. Introduction. We are concerned with the approximate solution of linear least-squares problems
FRACTIONAL TIKHONOV REGULARIZATION WITH A NONLINEAR PENALTY TERM SERENA MORIGI, LOTHAR REICHEL, AND FIORELLA SGALLARI Abstract. Tikhonov regularization is one of the most popular methods for solving linear
More informationA fast nonstationary preconditioning strategy for ill-posed problems, with application to image deblurring
A fast nonstationary preconditioning strategy for ill-posed problems, with application to image deblurring Marco Donatelli Dept. of Science and High Tecnology U. Insubria (Italy) Joint work with M. Hanke
More informationLinear inversion methods and generalized cross-validation
Online supplement to Using generalized cross-validation to select parameters in inversions for regional caron fluxes Linear inversion methods and generalized cross-validation NIR Y. KRAKAUER AND TAPIO
More information15 Singular Value Decomposition
15 Singular Value Decomposition For any high-dimensional data analysis, one s first thought should often be: can I use an SVD? The singular value decomposition is an invaluable analysis tool for dealing
More informationArnoldi-Tikhonov regularization methods
Arnoldi-Tikhonov regularization methods Bryan Lewis a, Lothar Reichel b,,1 a Rocketcalc LLC, 100 W. Crain Ave., Kent, OH 44240, USA. b Department of Mathematical Sciences, Kent State University, Kent,
More information2017 Society for Industrial and Applied Mathematics
SIAM J. SCI. COMPUT. Vol. 39, No. 5, pp. S24 S46 2017 Society for Industrial and Applied Mathematics GENERALIZED HYBRID ITERATIVE METHODS FOR LARGE-SCALE BAYESIAN INVERSE PROBLEMS JULIANNE CHUNG AND ARVIND
More informationTotal least squares. Gérard MEURANT. October, 2008
Total least squares Gérard MEURANT October, 2008 1 Introduction to total least squares 2 Approximation of the TLS secular equation 3 Numerical experiments Introduction to total least squares In least squares
More informationDownloaded 06/11/15 to Redistribution subject to SIAM license or copyright; see
SIAM J. SCI. COMPUT. Vol. 37, No. 2, pp. B332 B359 c 2015 Society for Industrial and Applied Mathematics A FRAMEWORK FOR REGULARIZATION VIA OPERATOR APPROXIMATION JULIANNE M. CHUNG, MISHA E. KILMER, AND
More informationOn the regularization properties of some spectral gradient methods
On the regularization properties of some spectral gradient methods Daniela di Serafino Department of Mathematics and Physics, Second University of Naples daniela.diserafino@unina2.it contributions from
More informationEstimation theory. Parametric estimation. Properties of estimators. Minimum variance estimator. Cramer-Rao bound. Maximum likelihood estimators
Estimation theory Parametric estimation Properties of estimators Minimum variance estimator Cramer-Rao bound Maximum likelihood estimators Confidence intervals Bayesian estimation 1 Random Variables Let
More informationInverse problem and optimization
Inverse problem and optimization Laurent Condat, Nelly Pustelnik CNRS, Gipsa-lab CNRS, Laboratoire de Physique de l ENS de Lyon Decembre, 15th 2016 Inverse problem and optimization 2/36 Plan 1. Examples
More informationLecture 2: Linear Algebra Review
EE 227A: Convex Optimization and Applications January 19 Lecture 2: Linear Algebra Review Lecturer: Mert Pilanci Reading assignment: Appendix C of BV. Sections 2-6 of the web textbook 1 2.1 Vectors 2.1.1
More informationMultigrid absolute value preconditioning
Multigrid absolute value preconditioning Eugene Vecharynski 1 Andrew Knyazev 2 (speaker) 1 Department of Computer Science and Engineering University of Minnesota 2 Department of Mathematical and Statistical
More informationDUAL REGULARIZED TOTAL LEAST SQUARES SOLUTION FROM TWO-PARAMETER TRUST-REGION ALGORITHM. Geunseop Lee
J. Korean Math. Soc. 0 (0), No. 0, pp. 1 0 https://doi.org/10.4134/jkms.j160152 pissn: 0304-9914 / eissn: 2234-3008 DUAL REGULARIZED TOTAL LEAST SQUARES SOLUTION FROM TWO-PARAMETER TRUST-REGION ALGORITHM
More informationChemometrics. Matti Hotokka Physical chemistry Åbo Akademi University
Chemometrics Matti Hotokka Physical chemistry Åbo Akademi University Linear regression Experiment Consider spectrophotometry as an example Beer-Lamberts law: A = cå Experiment Make three known references
More informationVAR Model. (k-variate) VAR(p) model (in the Reduced Form): Y t-2. Y t-1 = A + B 1. Y t + B 2. Y t-p. + ε t. + + B p. where:
VAR Model (k-variate VAR(p model (in the Reduced Form: where: Y t = A + B 1 Y t-1 + B 2 Y t-2 + + B p Y t-p + ε t Y t = (y 1t, y 2t,, y kt : a (k x 1 vector of time series variables A: a (k x 1 vector
More informationMultivariate Distributions
IEOR E4602: Quantitative Risk Management Spring 2016 c 2016 by Martin Haugh Multivariate Distributions We will study multivariate distributions in these notes, focusing 1 in particular on multivariate
More informationUnfolding Methods in Particle Physics
Unfolding Methods in Particle Physics Volker Blobel University of Hamburg, Hamburg, Germany 1 Inverse problems Abstract Measured distributions in particle physics are distorted by the finite resolution
More informationGI07/COMPM012: Mathematical Programming and Research Methods (Part 2) 2. Least Squares and Principal Components Analysis. Massimiliano Pontil
GI07/COMPM012: Mathematical Programming and Research Methods (Part 2) 2. Least Squares and Principal Components Analysis Massimiliano Pontil 1 Today s plan SVD and principal component analysis (PCA) Connection
More informationFrequentist-Bayesian Model Comparisons: A Simple Example
Frequentist-Bayesian Model Comparisons: A Simple Example Consider data that consist of a signal y with additive noise: Data vector (N elements): D = y + n The additive noise n has zero mean and diagonal
More informationINVERSION OF ELECTROCHEMICAL IMPEDANCE SPECTROSCOPIC MEASUREMENTS
BIOFUEL CELL POLARIZATION ESTIMATION INVERSION OF ELECTROCHEMICAL IMPEDANCE SPECTROSCOPIC MEASUREMENTS IMPORTANCE OF MODEL FORMULATION Rosemary Renaut, Mathematics, ASU Jakob Hansen, Jarom Hogue and Grant
More informationcomponent risk analysis
273: Urban Systems Modeling Lec. 3 component risk analysis instructor: Matteo Pozzi 273: Urban Systems Modeling Lec. 3 component reliability outline risk analysis for components uncertain demand and uncertain
More informationEmpirical Risk Minimization as Parameter Choice Rule for General Linear Regularization Methods
Empirical Risk Minimization as Parameter Choice Rule for General Linear Regularization Methods Frank Werner 1 Statistical Inverse Problems in Biophysics Group Max Planck Institute for Biophysical Chemistry,
More informationPseudoinverse & Moore-Penrose Conditions
ECE 275AB Lecture 7 Fall 2008 V1.0 c K. Kreutz-Delgado, UC San Diego p. 1/1 Lecture 7 ECE 275A Pseudoinverse & Moore-Penrose Conditions ECE 275AB Lecture 7 Fall 2008 V1.0 c K. Kreutz-Delgado, UC San Diego
More informationA Spectral Approach to Linear Bayesian Updating
A Spectral Approach to Linear Bayesian Updating Oliver Pajonk 1,2, Bojana V. Rosic 1, Alexander Litvinenko 1, and Hermann G. Matthies 1 1 Institute of Scientific Computing, TU Braunschweig, Germany 2 SPT
More informationOn nonstationary preconditioned iterative regularization methods for image deblurring
On nonstationary preconditioned iterative regularization methods for image deblurring Alessandro Buccini joint work with Prof. Marco Donatelli University of Insubria Department of Science and High Technology
More informationStatistics 910, #5 1. Regression Methods
Statistics 910, #5 1 Overview Regression Methods 1. Idea: effects of dependence 2. Examples of estimation (in R) 3. Review of regression 4. Comparisons and relative efficiencies Idea Decomposition Well-known
More informationFoundations of Computer Vision
Foundations of Computer Vision Wesley. E. Snyder North Carolina State University Hairong Qi University of Tennessee, Knoxville Last Edited February 8, 2017 1 3.2. A BRIEF REVIEW OF LINEAR ALGEBRA Apply
More informationPrincipal Component Analysis and Linear Discriminant Analysis
Principal Component Analysis and Linear Discriminant Analysis Ying Wu Electrical Engineering and Computer Science Northwestern University Evanston, IL 60208 http://www.eecs.northwestern.edu/~yingwu 1/29
More information